As the divorced single mother of 3 daughters, reading your post took me down memory lane. May I ask you if you have a relationship with his daughter? I'm not referring to the father and step mother dynamic, but do you have a one on one relationship with this now woman child? Have you ever?
And I do not ever recall hearing in ANY marriage vow saying to forsake your own kids. You have every right to spend as much time with your kids as you can/want. AND without GUILT.