You don't sound like an idiot at all batchel.
I think you made the right call on the visitation. You're right, you need to proceed as you are going to proceed from now on. No excuses for not adhering to his monitoring, as you have no doubt read here time and again, that's an old excuse. Now he may be absolutely telling it to you like it is, but there is history here and no reason for you to believe anything unless you have proof. Action, not words.
Hang in there! You are doing fine.