Is this what is best for the baby?
Is this what is best for you?
Is his behavior acceptable when he is there?
Is the baby, yourself, your heart, and your home safe when he is there?
Only you can answer, and there is no right or wrong answer. And, you don't have to answer those questions here, but in your own head. Trust your instincts.