I suggest starting with a 3461 log dump car with bin. Then get a log loader.
Logs are a size that a three year old can handle, and they are safe. Push a button to dump the load into the bin, and he can replace them on the car by hand. Six inch Lincoln logs would work well with this car.
Then get a log loader. Best is Lionel's old 164, last made in 1960. You can get one that looks reasonably good and works well for $70-$80 on eBay - there are several current buy-it-now in that range. Don't buy one that shows any rust in the photos and be sure that the seller states without reservation that both conveyor and dump chute are working. If the listing is unclear on this, ask the seller if he can assure you that the conveyor works well. These accessories are usually fixable with a scewdriver, pliers and WD-40 and that my include cleaning the motor.
The downside of the 164 is that it needs track on both sides about 15" apart. A longer and less attractive accessory that works alongside one track is the 364 log conveyor. You dump at one end and load from the other. Problem is that it needs 28" of straight track with control section at one end. The 364 is cheaper, but they don't look very nice.
In summary, I think log handling has good play value for a very young boy, and the accessoroes are not easily breakable. Later accessories my be more imaginative, but they tend to be more breakable.
