Sounds like the two of them have bonded.
You do have a say in who comes knocking at your door whenever, though not his talking to, enabling, or hanging out with your husband. It would not be out of line, IMO, for you to tell him that you would prefer not to have visitors after a certain hour, for example.
Sounds like your plan of getting away from it all, though, might be the most effective one.