cloudy, i would suggest next time you go to the "house" take the cops with you. let them see what he has done to the house and then have him removed. he is a threat and they will see this if he has torn up the house.
hold your ground, he is trying to emotionally manipulate you. even if you do love him, you still must take care of you!
you are doing fine and as you continue in your recovery you will see that you are fine.