You already have your answer, don't you?
What is your gut instinct telling you here?
Please trust it and if you are worried, don't let him back in.
You and your kids have a right to peace and safety.
To me the fact is is trying to use the children to manipulate you by saying
things like you won't let him back in is a huge red flag. Be careful!