so I called him and told him to get lost
You bet your sweet tail you did!!!! Good for you!!
Would you have done this before? Wow!
He caught you off guard so you said yes to dinner, that's almost expected. I would probably do the same darn thing myself. BUT, you thought better of it and without guilt over changing your mind you picked up the phone and toldhim to go away. BRAVO!!
His present lady is probably kicking him to the curb so he's calling you up and trying to get back in contact with you. This is his issue not yours. If he won't back off. Get in his face again and enlist the gym management like uncertainty said. You're taking that kickboxing class and you're getting damn good at it. He stood there and watched you. He ought to know you're no pushover anymore.
I'm so glad you told him to stay away. You are right, people are selfish and only thinking of their own needs. No one is going to defend your needs but you, so don't feel shy about stating them in an obvious way so they get the point clearly.
This is the growth you once could only dream of!!
Alice