I missed the part where you're upside down on the mortgage, but even so. Depending on how much you stand to be in the hole, I'd write some of that off. Look, if you were not living in the house, you'd still be having to pay rent to live somewhere. The real estate market may change, if you're planning to stay in the house.
I have written off several thousands of dollars "owed" to me for the sake of peace in my life. With a house and four cars, and your admitted ability to pay the debt yourself, I think I'd just lay claim to clear title for all the property and let him walk. It's not worth the hassle and expense of getting the money from him.
My experience/opinion.