IMO, his accusations of you being unfaithful are just manipulation and deflection by trying to put the blame on you for his actions. Don't buy into it, it's not worth your time.
Glad you are having a great vacation with your daughter. Good memories are priceless too.