Pouncer, I know EXACTLY how you feel. One of my goals over the past couple of years was to catch myself before saying something which might offend.
You can't beat yourself up over it. Just move on.
No matter what you say, someone will make it their mission to be offended by it anyway. I'm not referring to your joke - just in general.