You should not have to babysit him and watch over him. Have you spoken to him about your feelings or given him ultimatums about his using with you around or using at all? Only you know when you have had enough and the negatives outweigh then positives. I am curious-what has you going back time after time to see him when he lives far away? What are you getting out of this relationship?