As silly as it sounds, my friends and I used to pull this prank on each other in college. However, we never did it to our ex-boyfriends. You obviously still have a hold on him and he's not quite over you. I would tell him about the calls if you still talk to him? and say you are bringing it to the police just to scare him. Also changing your number is probably a good idea. I guess getting even is childish but leaving his number at the supermarket does sound pretty funny, but of course this would just add fuel to the fire.
Good luck to you