He may keep them from you for awhile but he will be back once his anger wears off and he realizes that without YOU, he may as well have NOTHING!
I don't think you did anything wrong by telling him to pack his stuff and go because as I was reading your post, I was thinking that if my son said that to me, after all I was doing for him, I'd tell him the same thing even if he took the babies and left.
You are not a door mat.