Yes, that is a good boundary. Boundaries are all about YOU. What is best for YOU. What doesn't matter is whether or not it is best for him. Setting boundaries is something we do for our own well-being while not setting rules for anyone else. He can associate with anyone he chooses, but then, so can you.