Yes--GO. Even if some parts of it make you uncomfortable, DO IT.
I had to force myself to have fun when I split up with my second husband, and then again when I was in early sobriety. It seemed wrong, somehow, to be doing it. You can't babysit him forever, it isn't good for either one of you, and my bet is you will be SOOO glad you went.