Afraid is not a word that is used in a healthy relationship. Have you considered that it might be time to end it?
I'm not sure from your post whether you're afraid of him? Toxic can cover a big range of behaviour, and if you have any fears for yourself please consider talking to a local domestic violence service.