As hinted in my comment, this is the topic of PDN (Power Distribution Network) analysis.
In general, we model the supply as an impedance, or when considering the ripple/noise between nodes on the supply, an N-port network.
simulate this circuit – Schematic created using CircuitLab
For the 1-port case, we have a source of (AC) supply current (here, M1 + Rpu), and the corresponding ripple voltage on VCC is simply the change in current times Zs.
Note that we ignore Vs for AC purposes, as it's a DC source set to zero for AC analysis. We're only concerned with the relative change due to Rpu's current being on or off (and how fast it changes between those levels), which we can further model as a Thevenin or Norton source.
simulate this circuit
For the general N-port case, Zs becomes a network of impedances between ports. There is the self-impedance (change on a node due to current through same node), and all the transfer gains, from current into a given node, to the voltage on every other node. And so on for each node, making a symmetrical matrix of coupling factors / impedances. (Or we can relate voltages and currents in other equivalent ways to the Z matrix hinted at here.)
(I won't go into detail of N-port network theory here, but suffice it to say, we can draw up node equations, specify such a network, and solve for the AC voltages in currents in the system, just as we would any other AC steady-state solution. The network basically drops in as a sub-matrix of the matrix used in nodal analysis.)
When we place a local bypass cap, we're simply putting that impedance in parallel with the supply port. There can be downsides. Consider if there is dominant stray trace inductance coming from the nearest node, and we place a ceramic capacitor here. Say it's 5cm trace length or about 25nH, and the capacitor is 0.1µF with ballpark 50mΩ ESR. Say the trace is coming from a very large ceramic whose value (C and ESR) can be ignored.
The resulting resonant circuit has a characteristic frequency of \$F_0 = \frac{1}{2 \pi \sqrt{L C}}\$ or 3.2MHz, impedance \$Z_0 = \sqrt{\frac{L}{C}}\$ or 0.5Ω, and Q factor of \$Q = \frac{Z_0}{R}\$ or 10. The resulting impedance at the feedpoint (VDD) has a maxima, in the same way that the series-resonant circuit has an impedance minima (with respect to its own impedance), and the max and min are reciprocal quantities related by the Q factor. Which is to say, 50mΩ ESR and 0.5Ω Zo give us a 5Ω feedpoint maxima.
We can simulate this quite easily:
simulate this circuit
For a 1A current stimulus, a ~14dB voltage response is 6dB less than 20dB, or half of 10V, or 5V, as predicted.
This is a rather contrived example of course, and indeed we might want to ask further questions if this was really the situation. For example if Vs is 5V DC, then the load must be about an ohm, and, well clearly the 0.1µF is inadequate for bypassing that, but also we would very seriously want to know the effect of the resistor itself upon the supply's impedance. Because when it's connected (M1 on), Rpu and Zs act in parallel, but when off, Zs acts alone. The effect is, if the switching happens faster than or comparable to the ringing frequency of the network, the resistance is PWM-averaged out and has a proportional damping effect upon it; if slower, then we can consider the transient case, looking at each edge independently. The important insight here is, because it's an open-drain signal, not a full-drive CMOS signal, the resistor is connected only some of the time, and we have a non-LTI system (in particular, it's time dependent: the AC value of Rpu is effectively switching between nominal value and open).
For other loads, IC supply pins for example, we are just as concerned about the PDN, but indeed it is harder to know what the actual supply current (AC/transient) is, as manufacturers never document it; rarely a hint is given regarding shoot-through current, and sometimes PDN calculations and limits are defined (IIRC, Altera (now Intel) FPGAs had such tools and calculations handy), but most often we are left to guess based on some vague recommendation ("at least so-and-so capacitance near the device"), which clearly is not advice that can be used in isolation (that capacitor needs to be connected to others, and so on in turn; what inductance and ESR is acceptable in those connections!?).
The best solution then, given that we are given such paltry information, is to follow that manufacturer recommendation -- put in whatever miserable caps they say to, and ensure they are well damped by following the PDN analysis. In other words, the components they say to place are (perhaps) necessary, but not sufficient in and of themselves. We are responsible for the extenuating circumstances, and ensuring we haven't created a pathological PDN.
Here, if we simply add some ESR to Vs, the resonance is squashed right out, and a comfortably low impedance is had at all frequencies; we might add ESR by explicitly doing so (for the bulk ceramic, put a low-value resistor in series; beware ESL of the resistor, it all adds up!), or using a type chosen for modest ESR (aluminum polymer, electrolytic and tantalum are typical options, check datasheets for ESR ranges). Damping can also be introduced in parallel, such as using a C || (R+Cs) motif, where Cs > 3C and R = Zo. (The latter is especially suitable for strongly-filtered local supplies, like a sensitive analog subcircuit might require; a large inductor (some µH say) might be used, and then bypassed and dampened with this.)
To Summarize:
Why don't we use decoupling capacitors for open-drain outputs?
We do, when we need to.
It may be that the PDN impedance is already low enough that local bypass is not required, given whatever the tolerable noise level on the supply is. This is more likely the case for large-value pull-ups, where the change in current is small, and for slowly-changing outputs where the pin dV/dt and therefore supply dI/dt is small (when supply inductance is a concern).
In the other extreme, load current might be so large that it calls into question, not only the local supply, but further extents as well, or indeed the whole circuit itself. Consider the case of a pulsed load (say it's a powerful LED, motor, etc.) in a portable device: the battery voltage itself fluctuates The impedance of the battery limits how much power can be drawn, while maintaining circuit function (keeping supply voltage above minimum operating level), or safe operation (e.g. to avoid exceeding minimum battery voltage). We can consider a battery an enormously larger equivalent capacitance than anything we would put in circuit as such, making this a fairly extreme example.
In general, we need to consider not just current, but time as well, i.e. the rate of change in current, and its duration (charge). The supply impedance is just that, an impedance, and it can be a nearly arbitrary function of frequency. We should generally design constant-voltage supplies to have flat and low impedances, peaking no more than a limiting value, for best results.
