This is one of the very situation why supervised visits are necessary. It's not about you and it's not even about him...it's about what is best for your child and a 19 month old child can never safely be left with someone who is not responsible in every way.
Please don't feel guilty, you are doing the right thing. Yes, it's sad it has come to this but it is what it is and it's up to you to protect your child...and it sounds like you're doing a pretty good job of that.
Hugs