I feel for you. However, try and convince yourself that NOBODY can rob you of your joy. Regain the control. You have the power within you to do it. Be happy and let his negativity be HIS own problem not yours. Nobody can rob you of your joy and serenity because it is YOURS and YOURS only. When my exAH does this (because he wants me to feel as bad as he does), I feel compassion for him and rejoy that I am not feeling the way he does.
Big hugs to you! We are all on the same boat here!!!