It's about you and where YOU want to draw the line. You cannot control your husbands behavior. Only your own and what you are willing to live with.
It kind of sounds like you might be drawing boundaries with no consequences.
I told him to tell one I don't want to see him around here again. There are 3 children that live here, I will not have them put into danger.
That is a boundary. Now you have to decide what you are going to do if it is violated (are you going to call the police? are you going to take the children and leave?....) If there are no consequences for violating boundaries, they become meaningless threats that make the problem worse.