Well, you've told him how it is. That doesn't mean that you were being mean. You were just being honest. And there is nothing wrong with you sticking to your boundaries and following through on what you know is best for you and the kids.
Don't start second guessing yourself now.
I think you're doing just fine.