If I were you, I'd tell management right now that he isn't welcome and to call the police if he refuses to leave. Let them be the "bad guys"--you can even tell them that you're afraid he will try to make you feel sorry for him and that you need their help.
These days, apartment complexes risk liability if they negligently put a tenant in danger, so my bet is that if you tell them that, they will be diligent about keeping him out.