Margaret, how long are you going to let him come and go at his own whim, while you are looking after his child? He seems to be using his home as a convenience.
You said you were 'scared' to answer the phone. Has he become violent or threatening in the past?
Do you have any local support like family or friends?