I like what others have said about if it bothers you it’s a problem. One thing you might want to know is that some are now looking at porn as a potential addiction. I've done some research on the matter and I've learned some interesting things. Pornography use is just recently (within in the last five or six years) becoming a research topic for those who study addictions. Just like a lot of other things (drinking, gambling, even gaming), it appears pornography use can develop into a compulsive addiction. When someone is addicted to something there are changes in the brain...those same changes are seen in compulsive pornography users. Some people call it a "natural addiction," much like over-eating, because it messes with chemicals (primarily dopamine, if I remember correctly) that are naturally produced in the body. However, just like food or alcohol or video games, it doesn’t develop into an addiction for everyone. If an addiction (or unaddressed addiction) in your relationship is a deal breaker, you may want to figure out if this is just a recreational thing for him or if it’s developed into something he can’t stop.
Also, from what I have seen and heard, your feelings of inadequacy are completely normal. It’s not uncommon for men who compulsively use porn to have a hard time connecting sexually to real live women. It makes sense that this would feel rejecting to you.
This is a difficult thing to deal with and I wish you the best as you figure it out. Good luck!