You've done nothing to be ashamed of. This guy sounds like a creep. AND he "likes it rough"--code for he's a potential rapist. Not necessarily, but potentially.
I suggest you let this one go and find a really NICE guy who respects you for who/what you are. You are unlikely to get that from this guy. And it doesn't matter whether he's an alcoholic or a drinker or stone cold sober. He's not a respectful guy. You can do a LOT better and you deserve it.