I might be weird but I have a work-week regimen that I really have to follow in order to get sufficient rest and be able to perform at a high level at work. I don't go out in the evenings or nights on work nights, because not doing so helps me maintain peace and serenity in my life. It took me a long time to identify calm, quiet evenings with limited activity and early bedtimes as a personal need. But now, I don't start major projects or take care of big chores on these evenings, I don't accept phone calls after a certain time, and I don't go anywhere unless it is an emergency. If a friend called me up in a bind of course I would go, but there is no way on earth I would regularly go out of my way to provide transportation for another adult.
Managing MY life and MY responsibilities is ENOUGH for me to handle. And I don't even have children. Believe it or not, there ARE many people on earth I can be in a relationship with who manage their own lives well and therefore CONTRIBUTE to my health and well-being instead of being DETRIMENTAL TO my health and well-being.
Every person is responsible for his or her own life. And I have learned that my thinking, speaking, or behaving as though another grown adult is incapable of figuring out how to manage his or her own life really is (1) hurting me and (2) hurting that person. I used to go out of my way ALL THE TIME for other people. Changing that felt like I was being selfish, and sometimes still does feel like I'm being selfish. But I've also learned that my perceptions regarding this are distorted. Although it may FEEL selfish at times, taking the BEST care of myself I possibly can is NOT selfish.
It sounds to me like you simply do not want to be his Soccer Mom any longer. In that case, my advice would be: Stop picking him up. And don't worry about his reaction. Just tell him straight, "I do not want to pick you up any longer, please find another form of transportation." And walk away.