Child support orders take into account where the children are living. If he feels the amount is unfair due to the living arrangements the onus is on him to file a motion to change the support amount.
It is a COURT ORDER. It is your CHILDREN'S right to support, not yours. He doesn't get to unilaterally decide what's fair. You have ZERO to feel guilty about.