Change your number. Set up a supervised visit, if not with an official at least with some other 3rd party (aunt, grandmother, friend). You don't have to communicate with him. Thursdays at the park from 2-4 (for example) is the end of your responsibility. You do not have to communicate with him beyond that.