so this guy is robbing you of your inner joy and you are overly concerned about being fair?
How is his drinking and passing out every night fair to you? I know that lonely ,empty feeling.
With gentleness, i say you could toughen up a bit here and put yourself first for a change.