I don't think you're crazy. I think you're signing up for more of the same.
He's in his fifties. His children are adults. If the whole lot of them are determined to walk on eggshells to kowtow to the ex, there's nothing you can do about it. You could plan outings for yourself while your boyfriend ( not even worthy of man-friend, really) visits with these people. That's signing up for a lifetime of special days - alone. Some people would, I guess, if life with the partner was awesome. You describe his latest iteration as 'grumpy, inattentive, and not interested' in you or anything else.
I think you can do better. And by better, I mean I'd rather have my freedom than be emotionally tied to a fair-weather lover.