You have a couple of problems to deal with when feeding a physical switch into a circuit like this.
The first is switch bounce. When you close (or open) a switch, you can't depend on getting a single clean transition from low to high (or vice versa). Rather, as the switch closes it'll actually transition from low to high and back to low for some time--on the order of milliseconds or so. So, what you normally want to do is as soon as you see a pulse from the switch, you set a timeout and ignore all input from that switch for the next few milliseconds or so (exact time rarely matters a lot).
Secondly, you have a metastability problem. Since the switch is asynchronous to you circuit, it doesn't necessarily meet the setup and hold times required by your flip flops. To keep this from causing a problem, you typically want to run it through a synchronizer circuit, which typically consists of a couple of flip flops:

simulate this circuit – Schematic created using CircuitLab
In theory, even with this you could still run into a metastability problem, but the chances are reduced to a level that's typically quite acceptable (note: this isn't the only way to deal with metastability issues either--if you look around, you'll find others).