Repeat after me: "It's not my fault, It's not my fault..."
I know my limits and this is one of the reasons I do not have children-because I know I would not be able to handle anything like this. I am so grateful that I had the foresight and self-awareness not to have kids. Thank you for reminding me of my limitations (reminding me in a good way, so that I can set appropriate boundaries to match with my wants and my limits).
I hope you did get at least a little enjoyment out of seeing your grandkids. Maybe you enjoyed watching him while he was asleep? Please try not to feel guilty. Like me sister says, "NO GUILT! It's POISON!"