Rozied - do you ever practice setting small boundaries? Perhaps skipping a phone call, hanging up first, saying "I have to go. Bye" without waiting for him to finish his conversation.
Maybe with some practice you will feel strong enough to visit your son without risking your serenity...
((hugs))