The kind of universal adapter you have there is most commonly created with a 317 voltage regulator. Open it up and have a look on how it works. If it is one of these things (I have one too that looks the same, same specs, just different sticker) then the following applies:
The 317 needs at least ~10mA current to start properly regulating. At 4.5V that is 450Ω. That means you waste almost 50mW, I hope your power bill won't mind.
Apply that resistor and measure the voltage. If it is roughly at 4.5V then go on, if it is still at 10.5V something is really wrong. Normally the circuit of such devices has a voltage divider based on 240Ω from \$V_{OUT}\$ to the \$ADJ\$ terminal, putting in various resistors as you slide the switch (possibly also selecting different windings of the transformer).
Lets stay a moment here. Often single transformer versions of that thing have ~16V-18V output from the transformer. Assuming something is broken in the path from \$V_{OUT}\$ to \$ADJ\$ and just the ~587Ω resistor for the 4.5V select is active, the 317 would regulate to 10V-12V. Check if this is the case.
Back on track, when you verified that you get 4.5V out at 5mA load, you can be quite sure to not damage the toy by too much voltage (assuming you always leave the resistor in place). Now if you have a lab bench power supply that does proper current limiting, set it to 300mA 4.5V and connect the toy and measure current flow. Is there any time that the current flow touches the 300mA (and wants to go beyond it)? If that is the case it might be that the designers of the toy made an unfortunate decision: rely on the implicit power limiting "functionality" of the adaptor to limit things like inrush current. I myself accidentally blew up a toy CD player this way. This is especially likely since the original supply is unregulated.
So if you could determine (either by inspection of the toys circuit, or the lab bench power supply and measurement method) that it doesn't do this, I would say it is quite safe to use that universal power supply.
If any of those things above did not go as expected (or if you couldn't measure the current thing), then all I can say is: good luck, I hope you can find the toy elsewhere to buy again...