Hi Sheila, if your husband is able to be a good father to his son, that will be a huge benefit for your son. I would hesitate to remove him if your EX is a good dad. If there is reason to believe ex is drinking to excess while looking after him then that's different. If you do have concerns, would it be possible to discuss them with your ex?
My own marriage broke up and i know my children were much less affected because we were able to stay on good terms.
There other factor is that by working part time you are possibly able to keep your son in a better standard of living that would otherwise be possible.