I am not real familiar with the Fastrack switches, but I did sort of know they are (or can be) set up for non-derail, which is good.
In that case the two switches for the reversing loops should be self operational when wired for non-derail. I am thinking a switch controller could be optional with these two.
Then you could combine the two crossover switches to each operate together from their own separate switch controller. That would get 4 switches down to two switch controllers.
That leaves the lower left siding switch and the upper right siding switch. I am not exactly sure what to suggest with those? I suppose they could be wired into their respective crossover switches, but I am not certain how that should all work together?
Maybe GEYSERGAZER will have some suggestions for those. He's much more knowledgeable about train operations than I am.