you are making this WAY too easy on him. you took a break on the proceedings to give HIM time to sort things out, which he is NOT doing. you allow him to spend the day at your home and he takes advantage and DRINKS. what you see as not being contentious he sees as free rein to keep doing exactly what he wants, without consequence.
HE is not in the least concerned about how any of this affects your child.....he is not motivated to do the right thing and keep the peace between you. his prime directive is to move back in and return to HIS normal.
if you want to move forward, YOU will have to take steps. even if they are unpleasant.