You're both custodial parents.
But law enforcement wants you to come to agreement, even though it is legal to cross state lines. They would probably stall you, and try to get you to find an agreement. I think police get uncomfortable in these situations and want people to work it out, even though it would be legal to cross state lines. They don't like being involved without an agreement formed before they disengage.
I suggest you try to work it out with him before he calls the police. Just reassure him that you will be back on time, as planned. But if he won't play ball...you may have to postpone your trip.
Here's the law, but it only applies to non-custodial parents, as per my understanding.
Child Custody Jurisdiction and Enforcement Act Summary