I have two teenage DD's. They bleated and moaned for weeks when I moved out and took up to my Mom's. Thing is, they are better off away from the chaos as I am. We are not going back anytime soon; I just bought a house for us to live in indefinitely. They are now mad about that. Oh well. I am the adult and I get to decide what is right for them (and me). I do believe this too shall pass.
Give your little one lots of patience, soothing, and understanding but if this is what you need to protect the well-being of your family, it is your responsibility to do this. And your first point...I left for the very same reason. Who will he blame when I am not around?! Well...he still blames me. Oh well. No longer my issue to have to face everyday (said with a huge sigh of relief).
What we have today is peace and serenity in our home. We will have this in our new home, too. I need this, my daughters need this, and I believe this is the right thing to do, regardless of whatever anyone else thinks or feels about it. Kids bounce back quickly, too. Once they feel settled and the sense of uncertainty passes.