My missus annoys me more when I am off the wine..... but in reality it is ME who annoys me more when sober because my real issues come to light... and deep down I have anger issues which when drunk are smothered by my dead drunken brain...
I suspect your ex/on again partner is venting his own internal anger at you because you are the only one there...... it may take him some time to realise this if he can at all
and as you said so... you cannot help him.. he has to help himself
If he is meant to be with you he will come back when you let him go