SMB........No, I do not think it was too harsh to ask him to leave....in face, it sounds. to me. like you have been willing to just accept bad behavior.
I highly recommend that you read the book "Co-dependent No More".....it is the most frequently recommended book on this forum. I think you will find that a lot of it resonates with you.