So sorry for your situation.
Please be sure to keep all emails and texts, you may need them at some point as evidence of his abuse (and it is abusive).
You may well be missing what was or what could have been, but you certainly aren't missing what you have now and that is all he has on offer.
Imagine your precious child being around him! What support do you have, where is your family? Have you sought out Al-Anon meetings at all?
In turn, he tells me I am using our daughter as a weapon.
Fact is, he is using your Daughter as a weapon against you. He never actually shows up for any visits you offer, just takes your contact as an opportunity to hurl insults at you.
As for his family, they only ever hear his side of things and they are his blood relatives. No support to be had there, I would certainly cut them off, block their numbers etc. His Sister must be very gifted to be able to diagnose a mental illness of someone she has never met!