hi jess-
your safety is the most important thing.
if you are scared to ask him to leave, then pay attention to that. do not rock the boat!
perhaps it's best if you leave, rather than asking him to leave. most of them don't leave anyway, they just dig their heels in.
i'm wondering if maybe you could take a holiday to your mother's house with the children? you don't have to make an issue of it, just merely tell him that you are going to visit with your mother. you could even invite him along because he won't go anyway as he can't drink and smoke there.
then, once you are at your mother's house, you will be able to make the phone calls to the domestic violence people and get your plan together.