I too am with Eddie here...I read it as you not falling for his quacking, not the other way around. But I hear your frustration at even answering the phone...getting involved in a conversation and having an emotional reaction to it. That's normal; feel it and then move on from it. 8 months into sobriety and I still have those moments...but now I recognize they are just fleeting moments in the grand scheme of it all and I can handle them better now.
Maybe next time, call his bluff and see what he does? You're trying to sell your house, right? So say "You can move back in here, but the new owners may not like that very much".