If he expects you to have full trust he's deluded Mrs sw did not trust me for way longer & rightly so I had no right to say forgive me & get over it if I did Mrs Sw would have left
maybe it's because he's 4 months and still has stuff to learn
Give him time but at the same time look out for you