I am starting to dig into the FPGA world. While exploring the FPGA SmartFusion2 architecture I found out that there is a microcontroller (ARM Cortex-M3) and at the same time a system controller. Why is there a need for both? Can't one of them replace the other?
The system controller is a group of dedicated hardware functions that manage the internal operation of the overall chip. It may or may not have a CPU, but in any case, it is not user-programmable.
In contrast, the primary function of the ARM processor subsystem is to execute application code.
That's not an FPGA architecture, that's a System-on-Chip that happens to contain a microcontroller and an FPGA.
Having a hardware MCU in there makes a lot of sense: in reality, systems containing an FPGA often need an MCU to load the configuration into the FPGA, and to do system maintenance tasks, which, would you not have the MCU, you would do by implementing a small CPU inside the FPGA and letting that run software.
Also, dedicated silicon implementing the exact same functionality as implemented in an FPGA usually is faster and uses less energy. So, if you just need your microcontroller to e.g. receive command packets via SPI, then extract the command from them, and set off the right calculations to be done inside the FPGA, you can save a lot of energy because much more components of your system can be put to sleep, and the ones that always wake up use less power.
The system controller is responsible for booting both the FPGA and the ARM CPU, among other things.
Standalone ARM CPUs also have similar logic to configure a minimal memory map and conservative clock PLL setting before starting the CPU core, which will then do the remainder of the setup.
In a combined FPGA/MCU fabric, this is usually extended to loading the entire application program from configuration flash, since we already need to do this for the FPGA tables, so loading a bit more RAM doesn't add much complexity.
Keep in mind that FPGAs need to boot really fast, for PCIe and USB applications where standby mode doesn't have the power budget to keep the memory contents, but the device is expected to return to full operation within a few milliseconds.
Booting the MCU and then having it shovel data from flash into the FPGA tables would simply take too long — this would start the FPGA PLLs after the CPU PLL rather than in parallel.