The PLL appears to be performing an integer divide, so the only significant parameter is the ratio between the reference clock and the output. Unless you're concerned about high bandwidth communication, the level of jitter which you can expect should be ittelevant to your application.
Unless the datasheet says otherwise, device ageing is not significantly affected by operating frequency. Supply voltage, heat dissipation - yes, 10% clock speed backoff not so much. Even overclocked you would be unlikely to observe a significant degredation over time in achievable frequency. Clocking of these devices is delay limited, not thermally limited.
To expand on the latter point, timing between registers is the limiting factor for power/area optimised designs. So in the M4 CPU, there are maybe up to 25-35 levels of logic (nand2) between typical pipeline registers. Depending on the technology, this determines fmax through propagation delay. Faster cores have longer pipelines (less logic in each stage), and do the same work over more clock cycles (trading throughput for latency).