He is trying to hold the baby hostage, because he knows its the only way to be 100% sure that you'll return from this vacation. Have you ever left the two of them alone together? Has he ever had even one full day being the sole caretaker? What if he did? If he's not drinking and you think its safe, what if you left the baby with him for a whole day? Give him what he says he wants, in other words, because he doesn't really want to be left alone with the baby. He wants you to cancel your trip as insurance that you won't leave him.