I think Severian and Dee both made good points. As they say "honesty is the best policy..." So maybe a heart to heart talk? See how your comment made him feel, and tell him your feelings too. The sooner you work it out the better chance neither one of you will be building resentments.
All the best......