Generally speaking, it's not easy to answer this question. There are SMPS regulators that have lower noise than the normal Linear regulators. The downside is they are more expensive.
If you need a low noise linear regulator that is more stable than normal linear regulatros, there are linear regulators designed for such purpose. These are usually used for analogue referencing and are limited in sourcing current. For example, LT3045 is an ultralow noise regulator that can source up to 500mA.
Notice that the feedback needs to be connected close to the output capacitor so the regulator can operate properly and reject the ripple and noise from the output.
The batteries are very good, as we can assume they don't have internal noise. This means if you are not drawing current from the battery, it's completely noise-free. However, if you are supplying a load, and the load is noisy (there is a ripple on current), then the internal resistor of the battery becomes very important. That internal resistance will create ripples on the voltage of the battery terminals.
This means the lower the internal resistance of a battery, the more stable the output. For example, a Li-Pol battery has a very low internal resistance compared to a Ni-Cad battery. So, for a specific noisy load, if you use a Li-Pol battery, you will end up with lower noise on the battery terminal.