The simple answer is you can’t tell until you try.
Some devices might ignore your attempts to provide an invalid clock, others might return broken data or spectacularly crash. Maybe you’ll shorten the expected lifespan of the part through overheating or another fault condition.
Some oscilloscope manufacturers are rumoured to do this with fast analogue to digital converters (ADCs). These are a significant portion of the unit cost, perhaps >£50 in a £500 device. The scope manufacturers test thoroughly and sort them into “bins” based on performance. This often works because the original chip manufacturer conduct their own testing and often label a part as a lower spec to ensure it works throughout the operating range (temperature, voltage, expected life etc.).
You might be able time get a generic answer about the viability of overclocking a particular type of device, e.g. SDRAM or a DAC or an SD card or an image sensor.
In the data sheet you posted are a few useful quotes, for example:
this A/D converter operates at speeds up to 12Mhz and is fully synchronous to the pixel rate
There is also a video timing generator built in. This suggests the capture device has sophisticated internal control circuitry that you can’t just fool with a faster external clock.
Regardless you’ll be using the device outside the specified operating conditions, so you shouldn’t rely on it without significant testing and error checking.