I have the same problem with the Fastrack 6-12054. I bought in new in 2016 and only use it every Christmas season. Each year when I set up the track the unloading button barely works. I've brought it in to my local Lionel dealer in 2017 and this year. He takes apart the controller and said the the contacts on the 4 brass tabs look like they arced and don't make a good connection. He has sanded then with emery cloth and also used a small wire wheel to clean the tabs.
The track is only used for about a month per year and the unload button is not used excessively or held down for a long time at once.
I've had the older 027 uncouple\unloading tracks and never had a problem like this
