He is doing what Alcoholics and Addicts do. Deflect, lie and manipulate, they are masters of these skills. They have selective memories, by choice, in an attempt to confuse and breakdown their counterpart, they do this with malice and forthought.
Keep your direction, your resolve, don't fall for his BS.
You deserve so much more don't let your self imposed guilt ruin your future, he just is not worth it...Right?