Even a power supply with the worst regulation won't output a higher voltage than rated when massively overloaded. You would expect an unregulated supply to output a higher voltage at no load, lowering to rated voltage somewhere in the middle of the current range and dropping to lower at full rated current. Beyond this, the regulation may become even worse, but normally the limit is a power limit - it will start getting hot.
If it is adjustable (you say it ignores the setting), then this is usually achieved by a linear regulator. For it to be unregulated and have a switch, it would need multiple taps on the transformer, which is expensive and not normally done.
So it sounds like your power supply may be broken. If you have any power resistors of between 20 and 30 ohms then you could perform a simple load test. They will need to dissipate at least a watt (if it is doing 4.5V at 0.2A), or you could parallel 4 or 5 1/4 watt resistors. Keep an eye on the heat, as if it is outputting 10V+ then much more power will be dissipated.
On the flip side, this could be an issue with your meter. DC brushed motors introduce vast amounts of broad spectrum noise onto power supplies. This can confuse meters - especially cheap ones. The worst case is for cheap RMS meters. You could be getting spurious readings.
However, it may be worth taking a step back. Are you actually intending on measuring resistance? The windings in the motor will have a resistence, but this needs to be measured with the motor stationary, ideally using a meter on the resistance range. But why would you want to know the resistance of a simple DC motor?
Are you actually trying to characterise the motor for some other purpose? In that case you really need to perform current measurements under no load, stalled and for several points in between.
