Jackthedog, something my solicitor recently said to me about contact issues with my nephew and his nutty aunt might be relelvant to you and your son.
She said that I had to show I was protecting him, and be able to prove that I was,because others will be doing their best to show I'm not and will have solicitors helping them prove it.
I know it's all too easy to get sucked into thinking 'the other one' (in my case this is the nutty aunt) should have some say, or some rights, and really the kids should see them because they are family.
Well, sorry but that's just not right. If it benefits the child to see them then fair enough, but if it's just to make us feel better that they still have family contact because they just should it's very wrong. As parents we have a responsibility for the childrens welfare, and their welfare should be paramount.
If there is any risk to your child, either physical, say from drunk driving, or mental, for example manipulation or abuse then please think twice about giving your ex so much unsupervised contact.
It may seem like the easier option, but would you ever forgive yourself if something happened that could be prevented by a bit of juggling and jiggling?
You'll find a way, there is always a way to get around a problem.