Moose,
Perhaps you can write the letter, to get your thoughts and feelings out. Much like a 4th step... but nothing says you have to send him the letter or share it with him. Sometimes, just purging ourselves of the angry feelings is enough.
I've learned that my relationships with various people are independent of one another, even if the people involved want to tangle them all together. As long as your grandson's mom is willing to have you in her life and the baby's, all is well . If that should change, then you may consider talking to a lawyer about grandparental rights, visitation etc. The good part is that you don't have to awfulize or worry about those things just yet.
Remember, what other people think about you is none of your business. Enjoy visits with the people you want to have visit, and snap some lovely pictures as you stroll on the beach.
Love and hugs from mom to mom.
Cats