I was just talking today with my sis-in-law about guilt, as it related to her mother who tries to give her an extra helping of it every time they talk. She does things more for herself now (including limited contact with her mother for her health and sanity) but allows herself to still take on the guilt.
I told her what I've learned about guilt thus far in my Al Anon journey - it's a hook. I suggested that the next time her mother makes a comment like 'you never spend time with the family' she can simply say 'I'm sorry you feel that way'. She doesn't have to explain herself or her decisions regarding her family participation.
He may try to make you feel guilty, but you don't have to take it from him. You can let him keep it. You've made your choice, you know it deep down, don't let someone else take that away because it's not theirs to take. If they don't like what it means to them they will try hard as heck to bring things back to THEIR normal, so that means they have an ulterior motive (consciously or subconsciously). Keep strong in your conviction - YOU are the best gauge of what you have to do.