I'm really sorry that happened to you.
Alcoholics/addicts will do anything to avoid taking responsibility for their own actions - including emotional assault.
Do you realize how ridiculous his statement is, though? If we had half the power over them that they claim we do, would we use it to be lied to, made fun of, yelled at, abused, neglected and ignored? Of course not.
He is saying whatever he imagines will hurt the most. If he is feeling distant from you - who moved? He did - by choosing addiction and mental illness over and over again. Stay strong - we are here for you.