Your kids will get over it, change your locks and go no contact with him.
My story is not with a brother, however, you are the captain of your own ship, his influence on your children is not good for them in the long run, they will carry their childhood into adulthood. It is your responsibility to set a good example for them.