loll the fact he's calling you a 'temptation' is a red flag. It suggests something he shouldn't be doing, is trying to resist, but gives in now and then. Sound like anything?
Your most sensible course might be to cool it and let more time go by. If you're not willing to do that, at least lower your expectations. And rather than him calling the shots, you might want to be unavailable now and then.