WTBH,
I think you did just fine. You stated your views, restated what you have said, and told him "...but thank you for telling me how you feel". and apparently without snideness or sarcasm. that's perfect.
I would only take issue with allowing the appearance that him hanging out is ok. He needs to drop the kids off, then leave, just like a non-custodial parent does. You can perhaps accomplish this by not being in the room when he does come in.