Based on your needs, yes, usb is ideal. Keep in mind, the U does stand for universal. You can have usb nics, wifi, mouse/keyboard/touchpad/touchscreen, sound cards, webcams, hell, even usb displays. It's not the most effecient, but for two nics, a temperature sensore, and a serial device, its designed for it. So much that the majority of computing device manufacturers rely on it heavily.** Not just cheapo sbcs or tablets, but even heavy hitters like Apple and Dell use it for touchpads and keyboards and webcams and wifi and bluetooth and ir ports in all their laptops.**
USB is ubiquitos, easy to implement, and has a very low board cost in terms of routing and parts.
While you can get 4 port arm devices, most have one or two root ports (buses), then connected to a hub ic. Your biggest bottle neck will be mixing High speed devices with Full or Low speed devices on a single hub. Cheap hub ICs have a Single Transaction Translator, that can bring a single low or medium speed downstream device to a high speed uplink. Better ones will have a MTT, Multiple Transaction Translators, so multiple low speed devices can be bridged at high speed speed. Or simply ensure that the low speed are on one hub while high speed on another.
But since you have arm as the intended SOC, you will have access to proper buses for sound or i2c or spi (for the temp sensor or serial) or video or even a built in ethernet driver minus the magnetics. Higher end ARM chips comparable to cheap Android tablets or routers are fully loaded.