Hi Lotus!
It doesn't sound controlling to me, but rather like you've identified a boundary. It makes me sad that you're scared to express it to him.
(though I have to be VERY careful as to how I word this, so he won't get defensive)
You can't control his reaction, only yourself. it's his gig how he responds to your boundary.
When I can do this well-sort out my boundary and express and hold it, it keeps me safe and (more) sane.