My perspective on this is, if you cannot speak your feelings, wants and needs to this man then yes, you need to tell him you can only be friends and that moving forward with marriage is not something you couldn’t consider right now, if ever.
When we feel the need to walk on egg shells to save a fragile relationship then it’s time to think about being in such a weak relationship or be ready to continue walking on those egg shells.