I understand how to hand calculate simple RC filter, but the differential input ADC usually employed differential anti-aliasing filter. How can I hand calculate or analysis this kind of differential filter? Moreover, how to analysis differential signal in general?
Such filters have 3 tasks: lowpass CM signals (usually noise), lowpass DM signals (the actual antialiasing) and prevent conversion of CM signals to DM.
the DM mode filter is formed by parts between In+ and IN-. It is an RC lowpass made by R1, R2, C1.
there are two common mode filters, one viewing into each input. E.g. for In+ you have R1 and then the network of the three caps to GND. In reality C2 and C3 will be badly matched. So do not treat them as having identical values! 'X2Y' caps will be very well matched internally, but you must use Class I types, otherwise the strong voltage dependence of Class II dielectrics will destroy the matching, whenever there is different voltage at IN+ and IN-.
you will realize that the CM filter is different for both inputs, if you assume that C2 and C3 can be unmatched. This leads to CM to DM conversion, which you don't want. One simple way to mitigate this is to make C1 rather large. You will realize in the formulas, that when C1 is large, the mismatch between C2 and C3 becomes less important. In fact, you will not do so much worse, if you have only C2 or C3.
As Tony cautions, these are 1st order lowpass filters, which will roll off rather slowly. If you are using a nyquist rate ADC, you need very sharp roll-off to prevent antialising from just above the legitimate signal bandwidth. If you are oversampling (if you signal bandwidth is far below the converter sampling rate) you can afford 1st order RC filters with slow roll-off. Also if you aren't oversampling but don't need very good antialiasing at the high end of the signal bandwidth, the 1st order RC is also fine.