You did well, and you will get stronger every day. It might be wise to just have the locks changed, he could cut another key before he gives it back and you wouldn't have to worry about it with new locks and new keys. Most landlords are fine with this as long as they get a key, some will even set up the locksmith and send you the bill when it's done.
It's sad that people have to be so hard to get along with. Thank goodness recovery teaches us that we don't have to attend every fight we are invited to.
Hugs