Some comments in no particular order:
1. The smoke motors I've seen have a DC-resistance of around 10 ohms. So when you say the motor current is about 50 mA, aren't you referring to the steady-state current when spinning? When you first start the motor, if applying 5V, the current "wants" to be 5V / 10 ohm or 500 mA. Of course the regulator, whether it be the 78L05 or your MCP1804 will immediately go into current limiting. The observation is the failures are occurring on startup (motor initially stopped) when the regulator is in current-limiting mode. This may mean absolutely nothing.
2. Several comments bring up "back-emf" from the motor as some kind of culprit. The back-emf voltage is of the same polarity as the applied voltage. And the magnitude of the back-emf is less than the applied voltage. So if you drive +5V DC into the smoke motor and quickly disconnect the drive while measuring the voltage across the motor, the back-emf will be some voltage less than +5V DC and dribble down to 0V in a second or so as the motor spins down to a stop. So back-emf per se does not generate negative voltages. Depending on perspective, back-emf actually helps you since as the motor spins up the current draw goes down.
3. What will generate nasty spikes is the inductance from the motor windings/coils. Since smoke motors are typically can DC motors, these use commutating brushes making/breaking contact typically 3 times per revolution. I agree with comments that spikes may be the issue. And since the so-called fly-back (not sure I've heard the term flash-back used) voltage from the inductance is proportional to the size of the interrupted current (V = -L di/dt), I can see how the condition is worse during startup when the motor current is 5-10 times the normal operating current.
4. GGG makes an interesting observation about the half-wave input. Note that the MCP1804 datasheet specifies an Absolute Max Output voltage of (Vin + 0.3V).
If during startup, the current is limiting at 200mA (per MCP1804 datasheet), then the input voltage at the 300uF capacitor is collapsing at a rate of about 0.7 V/msec (dV/dt = I/C). So in the 8.3 msec half-cycle when no track voltage is present, the MCP1804 input voltage drops by about 6V DC. I was imagining a potential scenario where the motor is spinning up (creating a back-emf voltage). At some point the positive back-emf voltage or 2, 3, 4 V DC might actually exceed the MCP1804 input voltage by more than 0.3V during the opposite half-cycle. But if you're confirming there is no input ripple at 100 mA, then this is probably not an issue. As shown in their diagram, the MCP1804 appears to use a FET which has its intrinsic diode from Vout to Vin; you've probably seen 78xx application notes where this diode must be supplied externally in certain circumstances.
So what to do. I'd send a question to Microchip/T.I. and get their expert opinion. Unlike the 78L05 datasheet which provides an equivalent circuit schematic of the regulator, the MCP1804 datasheet only gives a "functional block diagram" which is somewhat stylized making it hard to reverse-engineer what is really going on inside. The failure mode differences between CMOS (MCP1804) vs. bipolar (78L05) is a complex subject.
OTOH if you have nothing better to do and want to experiment, it appears to be an issue of how to keep the output voltage within the allowed range or practically speaking messing around with diode/TVS clamps and/or capacitors.
