I don't think it is controlling to say no. For all the reasons stated, your thoughts on this make perfect sense. He doesn't have to like it - this is your choice.
It's his choice what kind of job he takes, just as it is your choice not to continue to carry the financial burden. Seems fair enough.