There are lots of reasons why the pinouts are the way they are.
The easiest to tackle first is the power/ground pins. Advanced chips will arrange their power/ground pins to minimize inductance and to reduce the "loop area" of signals plus the signal return path. This will improve signal quality and reduce EMI/RFI. The absolute worst thing you can do for power/grounds is what was done on the original 74xxx series parts with power on one corner and ground on the other. Xilinx has a white paper on their "sparse chevron" arrangement that is interesting. If you search on their web site they have lots of other papers and presentations talking about it with actual measured results and stuff. Other companies have done similar things without all the hype and documentation.
For MCU's where most of the pins are user configurable there really isn't a good or bad way to do the pinouts (excluding power/grounds). It's almost guaranteed that whatever they do, it'll be wrong. It is very much like us buying a dress for the wife-- no matter what, it'll be the wrong size, style, color, fit, etc. You can either compensate in software by using different GPIO pins, or by creative PCB routing, or by uncreative PCB routing (a.k.a. just adding more layers).
Another possibility is that the pinouts have been optimized for routing the PCB on minimal layers, but you're not seeing that. CPU's, for example, that require connecting to a specific chipset (or RAM) often have their pinouts designed to make that interfacing/routing easier. This is common on things like Intel CPU's w/Intel chipsets. That's about the only way you can get two 800+ ball BGA's to connect together on a 4 or 6 layer PCB filled with other power/ground planes. In these cases there are often app notes that explain how to do the routing.
And the third possibility is that it's as simple as "that's just the way it ended up". It's similar to the "whatever we do, it'll be wrong" approach, so they just do whatever was easiest or cheapest. No real magic here or mystery here. In the past there have been chips that were popular but people would complain about the pinouts-- so years later another version of the part would come out that is functionally the same but with the pins moved to facilitate PCB routing.
No matter what, in the end "it is what it is" and we just deal with it. It honestly doesn't cause too many problems, and we're so used to it that it doesn't bother us (much).