"Look, lets take a break with the kids for awhile."....I don't speak to him that a way.
It took time for me to learn to speak with others differently. Keeping my communications in the "I" mode helps alot...."I'm taking a break..." and it lets the other person know what my plans are and also that their input is not required. By saying "let's take a break" the door is ajar for him to keep this issue on the table for further discussion. It's up to me to decide when a conversation is over.
I love this slogan: "Say what you mean, and mean what you say but don't say it mean." How the other person chooses to respond to my direct statement is entirely up to them.