The in and out thing is really confusing to children and its too difficult to explain to a small child. He needs to be there or gone. Is he at all open to talking about what is best for your son? Your son will not understand if you make the decision for him to go but your the mom and have to do what is best long term for your son. I wouldnt try to explain too much to a small child because they cant comprehend and i agree that nothing should be done in front of him.
A few years ago my son and I watch the movie Riding in Cars With Goys. In it the dad is an addict. When the son grew up he met his dad. His dad said the best thing he ever did in his life was to get out of his son's life. That part of the movie was extremely emotional for my son and I and really helped me to explain to him why his dad was gone. I know that your son is not old enough to understand that but have faith that in time he will. My son still says that that is our movie.
Sometimes having one good parent is better than having two when there is such confusion and pain.