Unless of course it's not just the current that matters, and that
voltage matters more than just producing a certain amount of current
In real circuits both matter. An IC, for instance, needs a certain minimum voltage to work (silicon diodes, for example, don't conduct at all below about 0.7V). On the other hand, too high a voltage may destroy semiconductor structures just by the force it exerts on the electrons. (Really high voltages can even cause arcs through normally insulating parts, permanently damaging the insulation. Much lower voltages can cause similiar effects through the miniscule insulating structures in ICs.)
Then, as per V=RxI, or I=V/R, higher voltage often forces more current through a given conducting structure, which technically is mostly not a problem by itself, but a higher current through any non-superconductor causes higher power loss in the conductor which heats the conductor and may ultimately cause irreversible thermal damage.
So, practically, we need to maintain the voltage between the lower and upper bound for the circuit to operate. Too low and it won't work, too high and it may burn out.
As others have answered, most non-trivial circuitry does not act like a constant (ohmic) resistance but varies its apparent impedance over time. Digital ICs (CMOS), for example, often briefly consume much more power on the edges of their clock signal than in between, so powering via some sort of constant current source will force higher voltage and too much current through them at times and/or not enough at other times.
The analogy with water (pressure=voltage,flow=current) holds: If you had a source that always forces as constant flow (current) of, say, 1 liter per second (Ampère) through your piping, what would happen if you shut a valve between the source and the drain? The source would increase the pressure until the flow is at 1 liter per second again, which may happen only after the (shut) valve or other parts of the piping break. Also, as the flow through all of the piping combined is fixed at 1l/s, flow and pressure in different parts of the piping will fluctuate in response to other (parallel) parts of the piping changing the flow through them, which is undesirable in electric circuits.
As to the question in the heading,
Why can't you just reduce the current instead of the voltage?
That's basically what we do all the time. Power supplies keep the voltage constant and modulate the current in response to the current the powered circuit draws (its resistance) over time to always keep the output voltage constant. The voltage stays at e.g. 5V even when you disconnect the circuit (=0A).