...and it's time for you to take care of you and him to take care of him. If your other kids feel so strongly about it, they can take him in. If they aren't willing to do that themselves they have no right to say a damn word to you. It's hypocritical and wrong.
Please consider going to an Al-Anon meeting. It will help you learn how to live the rest of your life with some kind of peace and/or serenity. Look here:
How to find a meeting in the US/Canada/Puerto Rico
Take what you want and leave the rest.
Cyranoak