I understand the anxiety you have about a job. When my business partners kicked me and my son out because of his drug addiction (Also one of the partners wanted the business for himself without us anyway) I wasn't sure how I was going to make it.
Do not worry about what your imagination thinks might happen. Stop those thought now. They will destory you. Do what anvilhead says and start now and be cordial. Build a better relationship, professional not personal. Good Luck and I hope it goes well!