Thank you for the responses! I am looking to maybe start applying for an internship and do some volunteering work so hopefully that will fill my time and be a good distraction.
I think I am going to take it a step at a time leaving them. By stopping going out with them, its just going to be difficult to explain why!