Standard CMOS ICs are always based on Silicon. Always. No exceptions. So not "mostly" but always.
Fabrication of an IR sensor is something completely different.
OK, they might uses similar fabrication steps but that will be it. The materials used will be different.
It is also possible to create an IR sensor in CMOS technology. It will have different characteristics than HgCdTe, InSb and InGaAs based sensors though.
But I have yet to see a CMOS IC manufacturing process where HgCdTe, InSb or InGaAs is used. In CMOS fabrication they want to limit the usage of other materials than Silicon as this might cause leaking PN junctions resulting in non-working circuits.