IMHO ..... In my humble opinion.
I am not in your shoes so it is hard for me. I would just tell him to go and have a good time and you'll see him when he gets back.
How can someone call you and tell you he loves you, then in less than 5 minutes get aggravated at you for asking to come along? I mean really, go back and re-read what you wrote.
Tell him to go and have a good time and have some peace in your life. Keep busy, do something and try not to obsess about him.