The only people who care that you do not drink, are the people that have a drinking problem. Sometimes they force a drink into your hand, or ask why you dont drink. Its common in asian countries to drink alcohol, its a sign of disrespect if you do not drink.