It all just sounds like it's too much drama and hassle - especially for such a 'new' relationship
Walk away - I think your being given a lifeline here. Just my honest opinion. He doesn't seem to make you at all happy. There was nothing positive you had to say about the relationship.