I think it does take time for people on both sides to heal.
But I odnt agree that its just your problem. If that was the case..then whats he so upset about? So to me that means he has a problem with it too.
I dont have any advice.
I know that we can only do what we can on our part. And sooner or later as long as you keep your heart and mind in the right place. Things will eventually come together.
I think giving him time is a great idea. And yourself as well.
Good luck and keep doing what your doing and hopefully after some time. It will all work out.