Hikeon,
His family paid for the unrequested visits and you want to be close with them.
I can't know all the details but this relationship with his family does not sound like a fundamentally healthy friendship. You cannot safely be close to him but they seem to want him to be close to you. It is just a guess on my part but they seem to be enabling his addiction - alot. And possibly endangering your sanity (if not your safety).
I am not suggesting cutting off contact with these mutual friends/family of exABF, but it looks like you have to set boundaries with those people too.
I've lived abroad for a few years. It was really great. It was amusing, sometimes a little annoying, being volunteered as the 'defender/explainer/apologist/fortune teller' (the DEAF) of US politics (and that was in the Clinton years). I hope your experience went well (apart from aforesaid situation). Happy early chinese new year.