The fact that you can only reach this man during working hours Mon-Fri should have stopped any relationship from forming, that would tell me there is another more important (to him) relationship in his life. I agree with the other cut off ALL contact with this person, you can't properly take care of yourself if you are trying to please someone else. I cut off all relationships (aside from family) when I got sober and it's only now after nearly 2 years of sobriety that I feel confident enough to start letting people back in. Take care of YOU.
Judy