Hi Amusic. It's your flat and you have every right to say who lives there and under what conditions. I have been like your ABF, and been kicked out on short notice on more than one occasion. It was inconvenient but not fatal. I got over it.
More than that I think that peoples refusal to tolerate my behaviour was instrumental in me eventually getting help. Seems to me you have been more than reasonable.