I wouldn't blame you for making the "she goes or I do" statement -- but only if you actually mean it and are prepared to go. He's shown no initiative whatsoever for actually doing anything about this situation, so if you really can't live with it, you might have to leave.