I am not a real RF engineer, so maybe you'll get a more authoritative answer if one of our RF experts comes by.
You are right, the power gain ratio should be given by \$|S_{21}|^2\$, so these two specifications at first don't appear to be consistent.
It looks like the explanation is they are measuring two different things.
In the S-parameter tables, you'll notice that the \$S_{11}\$ values are very high, for example \$|S_{11}|\$ is 0.889 at 500 MHz (for the first PN and test condition listed). This implies the S-parameter tables are for a measurement of the bare device, not the complete circuit with tuned matching filters shown in sections 6 and 7 of the datasheet. This is reasonable, since this lets you use the tabulated S-parameters to make a simulator model of the bare device in a tool like ADS.
The specified gain value (+17 dB nominal) is probably the gain with the assumption you provide an ideal matching circuit at the input and output.
If we look at the S-parameters, the \$|S_{11}|\$ of 0.889 implies that if we send 100 mW toward the bare device, 79.0 mW are reflected and only 21 mW actually usefully excite the transistor. The \$|S_{21}|\$ of 1.65 means that this nonetheless produces an output signal of 272 mW. So after accounting for input matching, the gain of the transistor is about 12.95, or +11.1 dB. A similar correction for the output matching effect should get you close to the +17 dB value in the characteristic tables.