You have a right to have your boundaries respected by your husband when it concerns your own safety and well being. Jettison the false guilt. Your hubs needs to grow up a little and figure out that he doesn't get to determine what's a big deal to you and what isn't.