GoodKarma,
Good for you to assess the situation and come to a decision. I know this can't be very easy for you.
How about teling him the truth, "This isn't working for me." Period. No explanation needed.
Good luck, check in and let us know how it works out.
Hugs, Strength...