Yeah, if it were me, I'd answer the phone the next time he calls. He'll surely apologize for not calling over the weekend and when he does, you can say (not in a mad way) that it's just as well because you've been feeling like things are moving too fast and your not really interested in a steady relationship at this time. You don't have to gripe him out for not calling, just take it as an opportunity to say something you've been wanting to say anyway.