By data processing inequality, by processing we are losing information. So why do we process the data?

Question

From data processing inequality(DPI) we know that if \$Z=f(Y)\$, then \$I(X;Z)\leq I(X;Y)\$. However, in the application, we process the data. For example, we pass the received signal via a matched filter to maximize the SNR. Isn't it in contradiction with the DPI?

Answer 1

Data processing helps to extract useful information from sources and also helps to merge different sources of information into one.

Lets say a transmitter emits a symbol sequence \$\mathcal{X}\$ and those symbols go through a additive gaussian white noise (AWGN) channel \$c(\cdot)\$ to give \$\mathcal{Y}\$.. At the receiver a filter \$h(\cdot)\$ is applied to \$\mathcal{Y}\$ to give \$\mathcal{Z}\$. \$h(\cdot)\$ is a filter designed to remove out-of-band noise and hence increase SNR (i.e because we know the specific frequency range that \$S\$ will occupy, all we are doing is designing \$h(\cdot)\$ in such a way that it just rejects all signal components that are not within the range of valid frequencies).

In this case we have

$$ \mathcal{X}\xrightarrow{\hspace{1cm}c(\cdot)\hspace{1cm}}\mathcal{Y}\xrightarrow{\hspace{1cm}h(\cdot)\hspace{1cm}}\mathcal{Z} $$

$$ \implies (S)\xrightarrow{\hspace{0.5cm}c(\cdot)\hspace{0.5cm}}(S + N)\xrightarrow{\hspace{0.5cm}h(\cdot)\hspace{0.5cm}}\left(S + \frac{N}{k}\right) $$

Where \$S\$ is the signal transmitted, \$N\$ is the noise added by the channel and \$1/k\$ is the proportion of noise power that lies in the same frequency band as our transmitted signal \$S\$. Because \$1/k\$ is a proportion we have \$k \geq 1\$.

Now if we start by looking at SNR alone then it is clear that \$\mathcal{Z}\$ has a higher SNR than \$\mathcal{Y}\$ because we have reduced noise by filtering with \$h(\cdot)\$.

$$ \begin{align} SNR(\mathcal{Y}) &= \frac{|S|^2}{|N|^2} \\ SNR(\mathcal{Z}) &= \frac{k\cdot|S|^2}{|N|^2} \\ \implies SNR(\mathcal{Z}) &\geq SNR(\mathcal{Y}) \end{align} $$

But for the information we have

$$ \begin{align} I(X;Y) &= H(X) - H(X|Y) = H(S) - H(S|S + N) \\ I(X;Z) &= H(X) - H(X|Z) = H(S) - H(S|(S + N/k)) \end{align} $$

$$ \begin{align} I(X;Y) - I(X;Z) &= H(S) - H(S|S + N) - H(S) + H(S|S + N/k) \hspace{0.5cm} \text{(1)}\\ &= H\left( S \middle| S + \frac{N}{k} \right) - H\left( S | S + N \right) \hspace{3.25cm} \text{(2)}\\ &= H\left( S \middle| S + \frac{N}{k} \right) - H\left( S \middle| S + N , S + \frac{N}{k} \right) \hspace{1.35cm} \text{(3)}\\ &\geq 0 \hspace{9.1cm} \text{(4)}\\ \implies I(X;Y) &\geq I(X;Z) \end{align} $$

The move from equation (2) to (3) is made by noting that because (S + N/k) can be obtained from the application of a function \$h(\cdot)\$ to (S + N), the knowledge of (S + N) also gives us the knowledge of (S + N/k). The move from (3) to (4) is made by applying one of the fundamental laws of information theory which is that conditioning reduces entropy or in other words knowing both (S + N) and (S + N/k) will enable us to extract at least as much information about (S) as someone who just has (S + N/k). This means that H(S|(S + N/k) \$\geq\$ H(S|S + N, S + N/k) which then proves the result above.

EDIT:

Let me try and come up with an analogy for you. Let's assume that you didn't know how any of the G20 presidents looked like. You literally knew nothing about them at all and also knew nothing about their countries. If I gave you a picture with all the G20 presidents and I asked you to identify the South Korean president you would have no option other than to guess and the odds of you getting the answer right would be 1/20. Your crude signal to noise ratio in this case would be 1/20 (\$\approx 0.05\$) (i.e number of south korean presidents / number of other presidents).

But if I told you that the president of South Korea looked asian then you could just cut off all the images of presidents that don't look asian from your picture and remain with about only 4 asian looking presidents. The signal to noise ratio of the resulting picture will then be 1/4 (\$= 0.25\$). So your signal to noise ratio has increased dramatically but just think about it, has cutting the images of the other presidents out added any new information to the picture? No it hasn't. The only reason the SNR was able to get boosted is because you knew the president was asian, so thats the information that makes it possible for the SNR to increase, not the actual data processing process of cutting the image out.

If you were to tell to two people that the president of South Korea was asian and then proceeded to give those two people 2 different images - one I give a full picture of all G20 presidents and one I give an image which has been been processed by cutting a number of presidents out of the picture - then the data processing inequality just tells us that the person with the full image has at least as much information as the one with the processed image (this is despite the fact that the processed image may or may not have a higher SNR).

In the case of filters in wireless communications, we are able to increase the SNR in that case because we know particular properties of both the signal and the noise (i.e we know the signal falls within a particular bandwidth and so we can cut out all data that falls outside that frequency band and in so doing increase the SNR).

Answer 2

Mapping f maps the content one symbol queue set to another. Matched filter does not handle symbols but signals that carry possible symbols + noise. Matched filter is a technical way to create a symbol queue where that f could be applied.

The mutual information in a process where a symbol source transmits symbols to some channel and a receiver regenerates those symbols (less or more succesfully) can be calculated from the production statistics and receiver's interpretation error statistics. A good matched filter can well help to make the mutual information greater when compared to a poorly designed. There's no contradiction with the DP.Inequality because the matched filter does not input nor output symbols.