Well part of my long story short was I filed for divorce and had her served while still living in the same house. A few months later, just before I was ready to file for sole use of the marital home she caved and moved out. That was a partiality rough period of time.
You have legal rights to 50/50 access of your son. I would get real aggressive on the legal front, sooner rather than later. The longer you allow this living/visitation arrangement the more difficult it will be to reverse it.