Call that lady NOW and get to a safe house. You are not to stay there a minute longer. This has gone on long enough. You must press charges. The emotional abuse alone is enough to get him off your property. He is TRESPASSING!!! Get your son and either run or call the lady and do something.
He now knows you will allow a punch to the face. This will escalate. I'm sorry, but it's going to get so much worse. Please get out now or get him removed.