Since you share children with this man, there are going to be times when you have to deal with him. If he isn't bothering you, and the kids enjoy him being there, why leave? It might be a bit uncomfortable, and certainly if he starts anything, you can always leave, but if it's just a matter of him being there and not causing any trouble, why ruin the kids' vacation?
Sometimes we have to just suck it up and be adult about it, even if it makes us uncomfortable. It won't last forever. I'd try to just not let it ruin the vacation.