First, realize that this circuit is how people created 5 volts 50 years ago. Now, people use switching regulators. If you needed a lot of current, the transformer was huge. I worked on an early computer where the transformer weighed 5 pounds.
For this circuit, for a first order approximation, where I is the load current, f is the line frequency, C is the input capacitance.
\$Vripple = \frac{I}{2fC}\$
You generally want a ripple of a volt or two.
For the 7805 regulator to maintain regulation, the input must be greater than the output plus the dropout voltage. For a 7805, the dropout is about 2 volts. So your peak from the rectifier minus the ripple must be always be more than 7 volts. Don't cut it too close, you want the minimum to be 8 or 9 volts.
The peak at the input will be:
\$Vpeak = 1.414V_{transformer} - V_{DiodeDrop}\$
As Finbarr pointed out, if your voltage is too high, you will waste a lot of power in heat. A 12 transformer output is too much. You want about 8 volts RMS.
A long time ago, 6.3 volt transformer outputs were plentiful because they were used for vacuum tube heaters. These generally would work, but maybe not under worst case conditions, not recommended IMO.
A hobbyist generally doesn't worry about the output regulation. These parts were designed to power circuits that require 5 volts +/- 0.25 volts. A typical 7805 can easily do this (provided that the input exceeds the dropout). If you really need to know, the 7805 line and load regulation specifications will tell you how much the output will vary.
