do you "sense" that he has all this angst inside of him, or does he demonstrate his anger in outward acts? is he directly unkind to you? you said he is cordial and kind at the weekly visits. i wonder as you two drift further and further apart if you are still looking for things you can FIX in him to keep him close and to keep yourself somewhat enmeshed?
bottom line, his stuff is HIS STUFF. and he will deal with it however HE deems fit to do so. every person has the absolute right to NOT deal with stuff. and they can drag as much baggage around with them as they can haul. and they can remain complicated messes of humanity.
HE may think he is doing just fine. regardless, unless you are being treated unpoorly BY him or you fear your child is in danger, then it just isn't your problem anymore.