First there's the obvious, like good line and load regulation. Line regulation tells you how the output voltage varies with varying input voltage. Load regulation shows how the output voltage varies is you vary the load. If you use an integrated regulator like the typical LM317 you should have nothing to worry about.
Ripple rejection tells you how input ripple (like from the smoothed rectifier voltage) passes through the regulator and shows at the output.
Do you need to go as low as 0V? Most regulators start at a bit higher voltage, like 1.2V for the LM317. The datasheet shows an application to start from 0V, but notes that full output current are not available at high input-output voltages.
Then there's power management. If you build a linear regulator like with the LM317 internal dissipation can be high. A 0-30V supply with 0-5A output dissipates at least 150W if you need the 5A at very low voltages. One way to circumvent this is to use a transformer with several taps, where the control logic switches to a lower transformer output voltage is less output is required.
Related is short-circuit current. A classical power supply will just limit the current to the 5A, giving the high dissipation I mentioned. This high current may damage your circuit too. Solution is a foldback current limiter, which decreases the output current when a short-circuit has occurred.
Switchers (SMPS, short for Switched-Mode Power Supply) are much more efficient than linear regulators and are more and more used in circuit, now that advanced controllers have simplified their design. But switchers are best at a given input and output voltage and output current. Their design for wide input voltage/output current range is more complicated. You can find them commercially, however, and you can recognize them by their light weight.