Sounds like your body may be dependent on it to sleep. I am 8 months pregnant and also take benadryl (sometimes) to sleep because my OBGYN recommended it.
I would suggest either quitting cold turkey, tapering off, or discussing it with your doctor before it gets worse. You may have a few rough nights of sleep when you come off but it shouldnt be TOO bad. Maybe ask about switching to melatonin??