You can question yourself and your actions, but the telling thing is his actions: if he only made himself available at night for sex, why? Someone who values you and wants to be with you will make the time, and not just give you leftovers. See this for what it is.