You answered because you are not ready to let go, and unfortunatly you are only hurting yourself, doesn't phase him one little bit.
Change his cotact # on your phone, to something like "Abusive Quacker" and then don't answer.
IMO,until you are ready to go no contact there really is nothing else to say.