Mary, can you pack his bags and change the locks? Don't do it if the consequences of it are not going to benefit you. Like if he gets physically agressive, or smashes anything, you do not need to go through anything like that.
Remind him and tell him he is not happy and it is best for him if he leaves. Is your baby OK? You need to think about the baby, too, do what is best for you and the baby. It takes much courage, but it is within you. You will come through this and be a better person, you are strong!!!
Laurie