How to convert from S-parameters to power gain?

With reference to the part below, which states 17dB gain at 500MHz for Vce = 12.5V

https://www.digikey.co.uk/product-detail/en/stmicroelectronics/PD55003TR-E/PD55003TR-E-ND/1664253

It's data sheet shows |S21| = 1.65, S21∠Φ = 34 for Vce = 12.5V

How do I convert between the two parameters?

You are right, the power gain ratio should be given by $$\|S_{21}|^2\$$, so these two specifications at first don't appear to be consistent.
In the S-parameter tables, you'll notice that the $$\S_{11}\$$ values are very high, for example $$\|S_{11}|\$$ is 0.889 at 500 MHz (for the first PN and test condition listed). This implies the S-parameter tables are for a measurement of the bare device, not the complete circuit with tuned matching filters shown in sections 6 and 7 of the datasheet. This is reasonable, since this lets you use the tabulated S-parameters to make a simulator model of the bare device in a tool like ADS.
If we look at the S-parameters, the $$\|S_{11}|\$$ of 0.889 implies that if we send 100 mW toward the bare device, 79.0 mW are reflected and only 21 mW actually usefully excite the transistor. The $$\|S_{21}|\$$ of 1.65 means that this nonetheless produces an output signal of 272 mW. So after accounting for input matching, the gain of the transistor is about 12.95, or +11.1 dB. A similar correction for the output matching effect should get you close to the +17 dB value in the characteristic tables.