I have been there many times thinking that if i go out he will to, or if i was late comming home and he knew that i was going to be late he would go "do his thing" And let me say that i have passed up many outings because i felt the same way you do.
so with that in mind i would like to tell you this is a time for your daughter and you to go and have fun and not worry because he will do what he wants and when he wants weither your there or not . I know its hard but you cant save him all the time live life for you not what if. I only say this because i have to passed up many times with my daughter because of the same feelings you have and i cant get that time back with my daughter.