He said he could have let me go to during recovery but didn't want to
Wow, BIG on him! The fact that he said that to you is one HUGE red flag. That's what you call a manipulation. It's meant to make you feel grateful that he didn't just throw you away and that you should feel fortunate that he's still willing to allow you to have sex with him. Like he's doing YOU some kind of favor.
This is your life, not mine, but I would walk away. Maybe down the road when things are healthier it might work.