I'm in a different country to you so the laws won't be the same. But I had similar problems with an ex. In the end I switched the access to a time and place that I felt was reasonable and made arrangements to be able to get to work etc. My ex took me back to court but the judge said that as he was getting 'reasonable access' then he was making a fuss over nothing. (And of course document everything! If possible get your boss to write a letter about the impact on your work.)