The point is, you don't HAVE to fight. This has been settled. Just because he doesn't like it and is bent on dragging you back into court doesn't mean you have to fight it. Just show up, calmly present your evidence and speak your piece.
Yes, I think you do have to accept that he will pull this kind of crap from time to time. Don't take the bait. You took it this weekend but you don't have to take it again.
If he presents any of that crap in court, you simply say you were abiding by the order, you were trying to point that out to him, but you can now see that the only way to deal with it is to come to court and let the court decide. Period.