Don't see how you can do it in the modulator, but it's trivial to do in the decimation filter. And the linked device does have a buil-in decimation filter.
If you do it in the decimation filter, does that
increase the input referred resolution? I'm not very familiar with the
mathematics of it... Because tables 2a and 2b clearly says that buy
increasing the gain you increase the resolution so it can't be a
simple digital multiplication. But maybe you average more samples and
get a higher resolution like that? But than why does the input dynamic
range decrease with the higher gain?
I think you answered the question yourself in this comment...
Digital filters tend to involve word growth : if you multiply two N-bit numbers the product is effectively either 2N or 2N-1 bits. (For signed data, 2N-1 bits is the practical result because multiplying 2 sign bits produces ... 1 sign bit)
The decimation filter (following the delta-sigma modulator) is generally an FIR filter. which involves multiplying some large number like 100 samples with 100 coefficients, and summing all the results.
If the samples from the modulator and the coefficients are each 16 bits, and you sum 2^7 samples, the result is thus a (31 + 7) = 38 bit quantity.
Now you need to extract a 16 bit result from this...
Word format (s = sign, d = useful data, n = noise) of the filter output...
37 ... 31 .... 23 ... 15 ... 7 ... 0
ss_sddd_dddd_dddd_dddd_dddd_dddd_ddnn_nnnn_nnnn
| low gain setting |
Above is a typical filter output for a large amplitude input
ss_ssss_ssss_sddd_dddd_dddd_dddd_ddnn_nnnn_nnnn
| high gain setting |
And for a low amplitude input
The "programmable amplifier" is simply a selector, choosing 16 of the (in this case) 38 filter output bits. The gain steps are constrained to be powers of 2.
By selecting lower order bits for the higher gain setting, naturally large input signals will overflow (thus dynamic range is reduced) but the last couple of bits contain noise, as shown in Table 2A.
This is just an illustrative example : the details of your device will vary but it is highly likely to follow this principle. The lower order bits (7 downto 0 here) will simply never be calculated, to save logic. And there may be saturation logic so that any positive overflow saturates at + full scale instead of wrapping to a spurious -ve output (and vice versa).
Reading the data sheet, this device apparently uses a much simpler Sinc4 filter instead of the FIR filter I described above (common in more critical applications like audio) but the principle is similar.
I have shown the top couple of bits unused : DSM performance usually degrades catastrophically near the top of the input range where the "noise" from noise shaping and the input signal sum to an out-of-range value. Simply ignoring the top bit or two is an easy way to achieve this. Quoting the datasheet for this device:
" The proprietary architecture used for the LTC2480 third order
modulator resolves this problem and guarantees a predictable stable
behavior at input signal levels of up to 150% of full-scale. "