Well, in a way, but your thinking is supported by his manipulation. He tells you that he "can't stand having people talk harshly to him" and yet he continues to make choices that tread all over people's boundaries. Also, what he can and cannot deal with isn't your responsibility. He's a grown man, and if he doesn't want to be spoken to in a specific way, he'll either walk away or do what it takes to build a relationship that doesn't yield that kind of talk.
I noticed my XAH also had this VAST inability to deal with stress of any kind. He complained about it constantly, and medicated himself with whatever was on hand, whether alcohol, sleeping pills, cocaine, videogames and movies, ANYTHING to distract himself from the normal stresses of life. The lack of coping mechanisms seems common among addicts and alcoholics.
In any case, reducing contact with your A doesn't make you like him. You would be choosing your serenity over anything else...and let's be honest, if you don't look out for yourself, who will?
There's nothing that says that you have to cut him out completely or permanently. You can do as you propose and reduce contact to a once weekly event, and see how it goes. Progress not perfection!