I'd think long and hard about wasting any time with this man. Just too much freakin' baggage.
I don't know that you owe him that much explanation--you haven't known each other that long. (If you had to look up his pic in the yearbook, high school doesn't count.)
I agree with LaTeeDa, you can just keep it simple and say, this really doesn't feel like it's right for me right now. Trust me, it's a lot less hurtful to break it off now than to let him down gradually.