Hmm.. I feel uncomfortable giving that sort of advice. But have you talked to him honestly? If you give him a chance, maybe he could understand your need for space right now. In any case, do whatever you need to do. Be your own priority right now - that's not being selfish. Just my humble opinion.
Matt