I'm a guy and I think he was using you. I'm really sorry to say that, but the tactic of always being too busy (except for sex) sounds like typical alpha male manipulation. Quite honestly I did that when I was much younger and in a very demanding job, and am very ashamed of those behaviors now. Reaching out to him more would have just made it worse. You handled it very well. You had nothing to do with this except you naturally (and honestly) started to get close and want more and he ran the other way because that's not what he wanted. When he said he wanted to remain friends he most assuredly meant "with benefits".