The points you make are absolutely valid. What you missing are some details that you could get by reading more on communication protocols. Here are some of them.
- The communication speed should always be chosen from the maximum required speed, not the one that can be achieved.
The reason for this is that with some exceptions (transceivers, buffers etc.) the data transferred must be either obtained or processed somehow. Input from human interface certainly works in completely different time dimension. And if your controller takes several seconds to process 1 MB of data, it would be pointless to transfer it at 16 Mbps speed.
- Because signal-to-noise ratio decreases with distance, the maximum achievable bandwidth decreases too.
There is term "bandwidth-distance product" often used in communication. This is another reason why direct MCU-to-MCU connections rarely use that high data rates.
- In modern MCUs the communication speeds are often independent from the CPU clocks.
For example, Xmega MCUs have peripheral clocks running at 2 or even 4 times the CPU speeds. Furthermore, controllers with USB interfaces often have their own oscillators.
- Various communication protocols are now supported in hardware.
Synchronous protocols like SPI (or I2C on slave side) have their clock signals coming from the different MCU. So, the hardware can use that clock to shift data to/from the buffer and only involve processor at the end of a message. More advanced MCUs with DMA support can even move the data between different peripherals or memory without involving CPU at all.
Asynchronous protocols like UART or CAN require synchronized clocks. They begin timing at start bit and then sample the inputs once approximately at the mid-clock point (UART) or up to three times at about 75% of clock pulse (CAN). Obviously the data integrity depends on the clocking precision. While CAN controllers can adjust their clocks using phase shift information, more simpler UARTs can not.
One common practice to achieve better UART synchronization is to use crystals with frequencies easily divisible to common serial baud rates. For example, instead of running aforementioned Xmega controllers at their maximum 32 MHz you can often see them with 14.7456 crystals, running at 29.5 MHz.
Regardless of the protocol, the combination of hardware buffering and DMA transfers make transfer bandwidth fairly independent from CPU clocks.
- Finally, when communication speeds go up it is more common to see separate controller chips rather than direct connection to MCU.
Not only that, but you can usually see parallel bus connection between MCU and high-speed transceivers like LAN or LVDS display. This is because the communication bus throughput becomes faster than can be passed in serial trough single MCU pin.
You've mentioned Ethernet in one of your comments. You should realize that with 1 Gbps Ethernet speeds no 16 MHz MCU stands a chance of processing that avalanche of traffic. For those speeds you should be looking at much more capable hardware, like used in RPi, for example.
By the way, this last point is just another form of point #1, i.e. if you cannot reduce data rate to something your MCU can handle, it logically follows that you need faster processor to deal with data flow.