The railroad geometry of E-Z Streets makes it hard to create tram style specialwork. One of the flaws of the E-Z Street geometry is that you get small gaps in the track when you try to create intersections. The solution requires custom cutting small pieces of track.
By comparison N gauge Kato Unitram was designed to create intersections. I will post a photo of Unitram later.
None of the systems is perfect. Unitram lacks single paved track. All its track is double track with a wider track center on curves (so you can't do a half curve) and a narrower track center at the start and end of curves and on straights.
At least we have something to work with.
Have you tried to put a small straight piece between the two curves of the switches and the intersection piece? It may take a little bit of work but adding a few small straight pieces like; two and a half inch curve to curve help with the alignment of the switches. E Z Streets is not an easy system to do fancy layouts with.
Lee Fritz