Yes, it's time. You are living in fear and living your life "around" the A. Obviously things are not improving so it's time to let him go and move on with your life. A restraining order should allow that to happen and make it easier for you.
I understand that you don't want to do it. We all have this desire to "save" and "protect" our A's - but I think that you need to save and protect yourself!!!
Please get the restraining order!!