I don't see how you are enabling paying him back you actually owe him. It's his choice and decision to do with that money as he pleases. If you hold back the money he'll try to hold that over you for leverage and things like even more money. Sounds like you can make a clean break at this point. Take advantage of it.