Stop thinking of his feelings. You have an RO in place and you know what that means. Follow the rules just as you expect him to follow them. The cat is your responsibility and you will make the decisions that need to be made. It's none of his business. Losing contact with the cat is just another consequence of his actions. Take care of yourself and your beloved kitty.