I used to drive myself nuts trying to understand why my daughter's father didn't seem to want to spend time with her. After a while, I enjoyed that he was absent. I know your son is quite young, but my daughter never did really like her father and hated to go on visitation. It got to where we both dreaded it when he did keep his visitation schedule.