I kinda see a red flag here. If he is in true recovery he should be honest enough withyou by saying that your drinking maybe compromising his sobriety. In my opinion maybe it isnt so much you but he is projecting onto to you what your drinking makes him feel like. Instead of being honest and saying that it makes him feel like wanting to drink he is turning it around on you. Instead of taking the responsibility for his inability to control his drinking he (sounds to me) like he is holding you responsible for it.
So I guess this is the choice you have to make. Do you want to feel that your drinking will be monitored in this relationship and babysat? Would it be ok for you to not drink around him? These are things you should be asking yourself for you NOT HIM. Is he even being honest with himself about all of this?
Just from reading your post it would seem to me that he has a problem not the other way around. So if its his problem and you have no control over how he thinks/feels/his recovery what are you gonna do for YOU????