Would it be possible to change the location of the party and just not tell him or his family? Do you have a friend whose living room you can borrow? Or is there a church you can borrow a room from? Or is the party at a rented place like Chucky Cheese?
And if they do show up, make sure you have a few of your friends on-hand as third-party observers to their crap. That may come in handy later (i.e., in court at a custody hearing).