You did nothing wrong except try to protect yourself and your child. If they talk about anyone it will be about him and what he put his family through. You should not be embarrassed for protecting yourself and your child. The neighbors will start to talk about someone else soon enough.