Not to be brutal, but continuing any type of a relationship with this man will be detrimental to both of your recoveries. Neither of you are in a frame of mind to be in this type of relationship. Quite evident by the fact that when you met up with him you went out for drinks. You must concentrate on yourself and your recovery or you might not have any life left to protect.
Hang in there.