He is trying to make you think you're the one with the problem. Your reaction is to question your own behaviors. This is 100% manipulation.
Don't REACT...instead, ACT!! Let him pack, leave, and then Change the locks !! You don't deserve that treatment.