I was going to get the locks changed, however I didn't want to give him a reason to go crazy.
Change the locks. It's perfectly acceptable and it is a wise decision. If he goes crazy over it, that's not something you have any control over. The only thing you can control is who has access to your house.