I don't know if you are in a community property state or an equity state. Either way, you have the kids and he has financial obligations. The court will impose child support. Unfortunately, "deadbeat dads" are all-too-common. The court can garnish his wages to force him to pay child support.
It sounds as if you don't have a job. This is your chance. Make it crystal clear to your attorney that you need to get training in order to return to the job force. The courts look favorably on a woman who has stayed home and quit working in order to care for her children. However, the arm of the law reaches only so far ... tons of guys end up living the high life while their wives and children go without.
Get your attorney to be tough as far as you getting back into the workforce or getting a better-paying job. This guy is a creep. His family sounds like they're supporting their "baby boy" so you are better off without dealing with this bunch of losers. Attorneys cost money - so sue him for your attorney's fees too.