I have two children with my XAH. He lives to try to trigger me. Until I got counseling and decided that no way was I going to let that happen anymore. It does not mean I fight back, b/c I don't. If I did, it would never end. I chalk his mumblings up to the BS that they are and move on. I have made it 100% clear that I won't respond to anything he has to say unless it is a CONSTRUCTIVE conversation about our children. Any other time, I refuse to respond. This has brought me lots of peace in my life. I could not care less what he says about me or to me actually. I communicate with him only b/c I have to, and as soon as my children are old enough, will never again. We too were together for 20 years.
As far as visitations in your home, I would end that if possible. If not, I would isolate myself and not be around him when he is there, as soon as your son gives sign he is done, visit would end and I would escort him to the door and lock it.
You deserve to take care of yourself. You have to make that time, and make it a priority.
Socially, you will have to force yourself to get out and have some fun. Once you do things enough they become habit. It's hard to change, but you can do it!
Big hugs!