you are NOT wrong. not one bit.
when you have been there, HE still did not stay sober.
when you have been there, he has not treated you well.
when you have been there, he left you.
your home is your palace, your special safe place. NOT a detox center. NOT a rehab facility. he's still using and abusing your good nature.
it's ok to let him go, again, for reals.