i am speaking from experience, walk, no run away and never come back. You already know he does not like your kids, what else do you need. Three years in an adult life is a short time, but three years in a child"s life is a life time.
Kids are not as naive as we sometimes believe. They are like little sponges. Their early years are highly impressionable and easily influenced. They need to feel welcomed 100% of the time. Sounds like your AB is not a good match for you at this time.