The frequency being high does not matter much, because depending on the PLL range mode setting, typically the frequency is divided down to 1-2 MHz for the PLL input. It also has a mode which can take up to 16 MHz input.
So from the PLLs point of view, it really does not matter if the HSE frequency is 2 MHz or 8 MHz or 32 MHz, the PLL will get 1 or 2 MHz anyway.
Everything else depends on which exact HSE you are going to use and how it is used. A ceramic resonator, a bare crystal, or an oscillator module, which could be crystal or MEMS based. The temperature stability, accuracy, power consumption, and jitter can be wildly different between different types of clock sources. Though jitter is relative, as dividing down a high frequency clock with jitter will also divide down the jitter, but usually it is not a problem anyway.
So you should focus on other aspects of the clock than frequency, such as physical size and accuracy, the actual frequency is secondary, as long as you can use it with the PLL to generate the frequency you want.
Alhtough having said that, it will be more difficult to find compatible higher frequency crystals with suitable parameters (load capacitance, ESR) that are guaranteed to be compatible with the HSE oscillator (gain margin), that it is to find compatible lower frequency crystals.