Do you have difficulty with the comprehension of time, or are you aware that you're going to be late but have difficulty making yourself be on time?
The reason I ask is because there's a less drastic version of fetal alcohol syndrome, one of the symptoms is an inability to comprehend the passage of time or to wrap one's mind around how long something is going to take and plan accordingly.
I highly doubt that my mom was utterly sober then entire 9 months she was pregnant with me, and I wouldn't be surprised to hear that I had issues related to that.
That being said, I don't have time issues in terms of being late. I tend to be ridiculously early - but then I sort of plan it that way on purpose because sitting in my car waiting is often the only time of the day/week that I get to be utterly alone and be inside my own head. So I sort of work in "me" time by showing up early to things.
For things like parties, I'm usually right on time. But then I'm one of those "overly responsible" ACoA's listed in the 13 common characteristics list.