No, you were not dumb for apologising. If you did something wrong then an 'I'm sorry' is appropriate.
If this loss of control is out of character for you, if the raising of your hand is out of character for you.. then I don't think you need any further proof that this situation is toxic and you should remove yourself from it.. and more importantly, your child.