I also believe that anger stems from elsewhere.
I was really angry at one time, for several weeks (at an ex). At some point I thought to myself, ok what is this anger doing for you?
Well initially it was really helpful, in fact it helped me separate from him (I had already been through the "sad"). I encouraged the anger, it was almost a break from the being down. Eventually I realized it was no longer working for me, I was getting nothing "from" it. It was no longer propelling me forward, it was just an emotion that was, now, annoying.
So I let it go. Didn't take long, in fact once I had that realization it was only a matter of a day or two before I was feeling better.
So, examine those feelings, what are they doing for you. I'm going to guess it helps to feel anger toward him as it separates you from him. It may be something you need to hang on to for a while until you are feeling "safer" with your other feelings toward him.
Yes, it's annoying but it's there for a reason. Once you decide to let it go you will really need to deal with whatever is causing that anger, but you will be able to.