Hi,
I have a slightly similar situation with a few people in my life, and what my husband and I have decided to do is just keep contact as minimal as possible. If there is a pleasant exchange we will 'reward' that will some phone calls or another visit. If the behavior is bad... a consequence occurs, which is usually no contact for what ever length of time we feel is needed.
I haven't completely cut off anyone... but I have stopped the emotional drain as much as I feel I can.
I hope that helps you... I think only you can decide how extreme or how lax to be on this. He can only do what you allow him to do to you and your son.
I'm wondering if you ever attend meetings for Adult Children? I think it would help to see how others handle this.
You are worth taking care of...and your son is being affected. That would be my breaking point and it was a few times when I saw my kids being affected by somebody's abusive behavior.
I'm sure that others will drop by with more ideas on this.
I wish you well.
cmc