I don't know if it's possible to be a few times a month, recreational heroin addict. I've heard that same excuse before and I'm sure a lot of other people have, too. First, most likely it's not just happening when you notice...I just doubt you're around to notice it all the time. You can't watch him every minute. Sometimes my Abf would pretend to be asleep when I left for work but would really run out the door the second I left to go get high so that when I got home 10 hrs later it wouldn't be that noticeable. Or he'd wake up at 5 am on the weekends and do it while I was still sleeping. I only found out after he started recovery that he was doing it everyday for months at a time but I only noticed a big enough change to call him on it every once in a while. Also, the amounts he's doing at different times may have an effect on whether or not it's clear to you.
Second, even if he is only doing a few times a month that's not ok because eventually he's going to need to start doing it more often. The fact that he's withdrawing large amounts of cash is a big warning sign. Make sure to protect your finances. You're married so I don't know if you have some kind of joint account, but be careful. You don't want your money going towards his habit