How about writing down what you'd like to say to the pastor, stashing it away for a month or two and then taking it out again at that time to see if you still feel it is necessary to communicate the information? If you think that this is something that could in the long term "heal" you, then I'd say go for it.