You are on the right track, here. CN.
This is something that limits your life, your growth... There is no need to give him a reason.
You dont want to enable his irresponsibility for himself anymore at your own expense.
He will probably not like it.
This is a good boundary. I can relate to feeling scared. But this is the beginning of a path of taking your own life energy back.
Its yours.