Maybe it would be a good idea to not answer your phone in the middle of the night?
I know that I don't. I don't have kids, so there is NO ONE who I am worried enough about that I would take a call from them at 2:45AM, especially from an unknown number. I leave my phone on mute, in another room, when I am sleeping.
And honestly, if you weren't near this guy when he said you were, then I would just ignore the whole thing. He would need some kind of proof -- photo or video, or the police would need to see you there, in order for him to be able to make it stick in court.
Either he's trying to make drama, or you're not leaving him alone. In either case, no contact and minding your own business is the best solution.