You're not going crazy. Porn can be a real addiction. Sex addiction is a serious problem. I have an ex boyfriend who was a sex addict. Believe me, it can be very destructive. He had an addiction to phone sex, and had hundreds of dollars a month worth of bills.
Don't take it personally. It's nothing against you. I went to a counselor at my college and she helped me understand some of what was happening.