Some people are sensitive to caffiene. If it affects you negatively, I'd say you're wise to give it up. I know since cutting coffee out of my diet (about a month ago) my blood pressure has lowered back to normal.
As for abstaining because it's a drug? Eh... your personal choice and reason for doing so is all that matters.