As far as I can see, you don't have any power supply decoupling capacitors anywhere on your collection of breadboards. It's no wonder it's "touchy" and sensitive to glitches.
You should put about 100 µF (electrolytic) on each one of the power strips in your setup, and somewhere in the range of 0.1 to 1.0 µF (ceramic) directly across the power pins of each chip.
Even though you're running at glacially slow clock rates, the "edges" (transitions) on the signals still happen very quickly, and cause spikes of current to be drawn from the power rails that can easily upset nearby circuits. This is exacerbated somewhat with the relatively large parasitic capacitances associated with breadboards — PCB construction would be much "quieter", but you'd still need decoupling.
Another bad practice that I'd recommend that you stop is putting bare LEDs across logic outputs. At least use some series resistors to limit the current to a few mA. Without them, you're severely limiting the voltage range of your signals, which will cut into your noise margins and lead to hard-to-diagnose problems later. Better still, use buffer chips to drive your LEDs.