yes, you should be concerned. Is your son in therapy? Are you? I think its time to bring in professional 3rd parties into this scenario so that there is awareness and guidance as to what to do.
Saying this gently: I'm reading some denial and minimizing in your post which means it's hard for you to accurately assess the safety of your situation. Please talk to someone.