Take a couple of steps back to energy, this will allow you to clear your head of the details of turns, volts, inductance etc.
When the magnet is on, and operating at its design flux, there will be a certain amount of energy stored in the magnetic field.
If you are happy to change the field slowly, so change the amount of stored energy slowly, then you only need a small amount of power. To change it quickly, you need a large amount of power. Power is rate of change of energy.
Power is volts times current. Now unfortunately, the copper wire has resistance, and this confuses the situation a bit, because we also need power to push the current through the resistance of the copper wire. However, let's simplify for a moment and imagine that we have a superconducting solenoid, like an MRI magnet. Just like a spherical cow in a vacuum beloved of physicists, this allows us to concentrate on the important bits.
To change the field quickly requires a lot of power, so more volts at any given current. If you alter the windings so you need more current to produce the field (reduce the inductance), then you will need fewer volts to slew the field at the same speed, and vice versa.
For any given electromagnet, you slew it as fast as possible by using the maximum drive voltage you can achieve. This might be limited by the breakdown voltage of your windings, or switching devices, or power supply. To slew the field up, you need to supply energy. To slew the field down, you are withdrawing energy, so you can generate the voltage by passing the current through a resistor, you do not need a power supply.
If the voltage range you can handle is limited, and you are not yet at your current limit, then you can improve the slew speed if you rewind your magnet to use more current, obviously limited by the maximum current you can tolerate. This will reduce the voltage needed to get the same slew power.
Finally, what about that pesky copper resistance? It means you have to supply power to keep the magnet on, and that your slew up takes longer, and your slew down is quicker. It does not alter the general conclusions that more slew power means faster field changes.