I completely agree with everyone about getting to a lawyer and letting them know what is going on. You can request supervised visits and/or no overnights. Document, document, document. Depending on the state and the judge you get, you may have to do more proving than you might think to claim alcohol abuse.
I'm curiously if your AH wanted your daughter to come over that particular night. Maybe if you made a regular schedule rather than an impromptu visit, he might be more inclined to be there? He is completely in the wrong for leaving her (and the brother is NOT an acceptable adult). Just thinking he had made plans but not wanting to say no to your daughter, he let her come over. Again, trying to think of ways to allow your daughter visitation with less risk.
I still think you need to talk to an attorney. They will let you know what the laws are in your state regarding visitation and the best way to handle this all. (((HUGS))) This is the part of divorce that I hate the most. It SUCKS to feel like the bad guy with your kids but ultimately, you are the parent and you have to do what you think is best for them until they can make those decisions on their own. Hang in there!