You have every right to ask him not to join you in Florida. Sometimes it's hard to tell people "no", but I think if you give it shot, you just might realize the other "no's" get easier in time.
One of my favorite expressions: "If I can't say no, then my yes is meaningless."