You are separated from him. You can decide when he comes and goes. He doesn't have the right to "decide to stick around" if you don't want him there. If you want to keep the scheduled "together" times, that is your decision, but you don't have to let him just hang around.