You need to ensure he feels loved and cared for, but you would not be helping him by having guilt over it – he must not act out physically on you.
I was not forced to any visits as a child, these cases are very difficult at least here.
I think psychologist are fine – if they are good.