If a DC coil was just a coil, it would saturate. If an AC coil was just a coil, the magnetic field would drop to zero at 120 Hz and the relay would chatter. Presumably the designs in some way prevent these problems. How? And does it prevent me from putting DC on an AC-rated coil?
The DC coil relay has a resistance from the copper wire typically used. The current is limited by that resistance.
AC coil relays have inductance as well (the DC relays have inductance as well, of course, but it does not affect the 'on' current). They also typically have a shading ring that acts as a shorted turn in order to cause a magnetic field 90° out of phase with that from the coil, so that the total magnitude of flux does not drop to zero at the zero crossings.
Edit: As Andy says, an AC relay will work on (greatly reduced) DC. Of course you can also make a DC relay work on AC by adding stuff to it (if you use a capacitor filter you might need only 18VAC (RMS) to operate a 24V relay, and 24VAC would cause it to overheat and fail early).
A Dc relay coil has resistance that limits the dc current. An AC coil relies on its impedance for governing the current. An AC relay will remain contact closed due to mechanical inertia and a little mechanical hysteresis and, the fact that an alternating north and south pole both attract the relay armature.
Putting dc on an ac coil should work fine but be prepared for the resistance to be low. In other words, If relay is rated at 24 volts ac don't use 24 volts dc.
In theory AC coils can be driven with DC as long as you limit the DC coil current to the level of the AC holding current (to keep the coil from overheating). For larger relays i.e. contactors, a DC current limited to the equivalent AC holding current is often insufficient to actuate the contactor effectively. Since the coil inductance of the contactor increases once the contactor is closed the AC holding current is less than the AC pull in current. DC contactors usually have some mechanism to shift between a coil pull in current and the holding current. This is the difference between an AC coil and a DC coil in large relays. The flux shunt (shader ring) is external to the coil and only shorts a portion of the core.
Recently, I had a problem with a batch of four 2-pole plug-in relays that had coils marked as 240V AC. When connected to the supply, three of the four merely 'chattered' and failed to 'pull-in' the contacts; the fourth operated correctly.
Believing the problem to be one of poor manufacturing quality control, I received some replacements from the same supplier which subsequently worked OK.
I assumed that the three faulty relays had been fitted in error with DC coils. Consequently, rather than binning them, I overcame the problem by internally fitting 1000 PIV DIL bridge rectifiers allowing the relay coils to operate normally on raw DC (i.e. no filtering).
Perhaps the use of 1000PIV devices was overkill, but I wanted the rectifiers to have a sufficiently high withstand voltage to the back emf generated when the relay released.
Additionally, as the relay coils were now operating on DC (albeit raw DC), I could have fitted a single rectifier wired in inverse parallel to the polarity of the raw DC to snub the back emf, thus permitting use of a bridge rectifier with a lower PIV, but a lack of sufficient space within the relay precluded this solution.