He's already pushing your boundaries to see if you'll cave- and if you do it will only be harder to keep any boundaries in the future.
Red flag!!
If he really cared about you, and not just his selfish needs, he'd respect your feelings on the issue.
Btw- not seeing you unless drunk or sex is involved - flaming red flag!!
Are you uncomfortable with the way it is going? Trust that.