It's different for everyone, but keep in mind that you are on day 4....possibly still in the physical withdrawal stage. For me personally, I think got over that initial WD hump at about a week or so, but every day got a little better.
The emotional side of things can take a lot longer...that's why it's so important to have a plan to deal with it.