Can you set up an hour-long visit at your house, and you nurse prior then head out for a coffee shop nearby? He can call you on your cell if the kids melt down.
One of my favorite sayings here is "more will be revealed." It reminds me to kind of trust that things will work out as they need to. This includes contact with kids. Your A will show what he's made of when it comes to seeking out time with the children. Mine was a real douchebag when he lived at home with us, didn't really have time to bother with the kids, and at one point my daughter described him as "an acquaintance" (sad). After I gave him the boot, he surprised me by turning into a much better dad in some ways, he really made it a priority to actually *do* things with our kids and have fun days out. They've all expressed that since our separation, they actually feel closer to dad, which is good. So trust that more will be revealed, and let your A show you what he will do.
Good on you for nursing and delaying solids! I've never had a moment of regret for the 6 continuous years I nursed (through several babies), and recently, my OB-GYN informed me that I'd reduced my chance of breast cancer by something like 90 percent due to that!