merrygoround, Welcome to SR! I personally would be more frightened by my child thinking that the lifestyle of a dealer/addict is okay, then why did I keep his father away from him. Certainly doesn't sound like your ex is going to change anytime soon (if ever), therefore, since you can afford taking care of your baby without the ex, I would.
Hugs,
Chris