If the second driver is powered down, you're almost certainly going to violate its maximum acceptable ratings.
There do exist ICs which are tolerant of moderately-high voltages wrt ground, independent of the power rail being present. But semiconductors naturally have parasitic PN junctions to the power rail, which would become forward biased in your scenario and risk latchup. The cheapest way to protect against latchup is to augment these parasitic diodes with intentionally-built protection diodes.
You would need a system that doesn't have either the protection diodes or the parasitic PN junctions. Not only that, you need the existing system to be already using such, which is highly unlikely. You could check the datasheet to be sure.
And note that all of the above is also true for input stages. Normally you can fanout a single low-impedance signal to multiple ADC inputs, but not when some of them are unpowered, as the protection diodes will cause clipping of the signal seen by all the rest.
In both cases series resistance can help.