He taught you what to expect from him. You don't want to be hurt again, so you can't be around that person. You're right. If he really takes it out on your sister, that's not your fault, he's just a nasty toxic person. And why the hell does he care if you visit him anyway? You and your sister can do stuff together elsewhere.