I know that the FPGA-DSP combination is typically used for high-end power electronics/ultrasound/MRI/etc. Is it possible for the soft-processor to fully replace the DSP even on lower-end FPGAs such as Spartan 3/6?
Added: What would be the reason for having multiple softcore processors in one FPGA?
Feel free to skim read, or skip to the end. I realise I did go on a bit!
Generally you wouldn't use a soft processor to replace DSP stuff. Dedicated hardware can generally handle higher volumes of data faster because you would design it to do a specific task very well, rather than being a general purpose CPU.
Where soft processors come into their element is control and coordination.
If you were to design a system which needed to process a large volume of data, lets say high frame rate image acquisition, it would not be possible to use a soft-core processor to handle all the data, there would simply be too much overhead in the CPU. What you would do is design dedicated firmware to do the specific acquisition task needed (e.g. filter the data, store to memory, etc.).
However you still need some way of instructing it when to do things - when do you want to be capturing, has the device been instructed to offload the data, etc. These things are not very easy to do in dedicated hardware, not if there are sequences of events with user input, basically tasks which do not do the same thing over and over again. In this case you would use a soft-core processor as it is far easier to write procedural code for some tasks.
Another (real) example, I have been working on a ultrasound acquisition system which streams data via PCIe. The tasks it does are communicated from the user and various parts of the system need configuring. The coordination of the system does not require large volumes of data, but instead needs flexibility, so it is well suited to a soft-core CPU programmed with in this case C. To do the same thing in physical hardware would require vast amounts of resources most of which would be used infrequently so would see no benefit compared to a CPU.
It's worth noting that some tasks may vary depending on user input, but are still better in dedicated hardware. In fact one part of the code (programming DMA controllers to store data on trigger) was originally done in the CPU in about 15 lines of code, but because that bit needs to be done the moment a trigger occurs, using a CPU which may be busy with other stuff is not ideal. The task is instead programmed into a Verilog module, but in the process becomes a massive 500 line state machine with about 15 states and a whole heap load of supporting logic - no really. But even though it uses up far more resources, it is time critical, so is a necessity.
Similarly I need clock cycle accurate trigger generation, so a module for performing that task is part of the system rather than doing it in a CPU. Both this core and the one above are examples of how you can use a CPU to do some tasks, but for other critical ones you can develop hardware to complement the CPU - in the same way you have timers, etc. in a microcontroller.
So to summarise:
FPGAs are great flexible tools, but most designs need a combination of soft-core CPUs, configurable modules (e.g. timers), and dedicated single-task hardware.
CPUs are great for user interaction, controlling the order of events, configuring controllers. They are like the coordinator, the brain.
Some designs may need to do some fairly repetitive tasks which can be configured to suit different inputs - timer modules, character displays, button debouncing, etc. These could easily be done with a CPU, but if you want to do several of them accurately at once it becomes more tricky - they are sharing the same CPU resources. So what you can do is move them into dedicated hardware which is closely connected to the CPU - give the CPU chance to do other tasks. These help the CPU do its job and interact with its surroundings, like its senses.
Dedicated DSP, data transfer (DMA) - basically any task which will do the same thing over and over again at high speeds - can really benefit from dedicated logic in terms of speed, and also possibly power. These are like the muscles of the device, the do all the heavy lifting.
You'll have to excuse the rambling on a bit, but I do like this field of EE. Hopefully the above is understandable and gives you some extra insight to the wonderful world of FPGAs.
As Tom mentioned, the MicroBlaze is not so much a matter of replacing a DSP, but replacing a traditional microcontroller that may otherwise be on the board.
This is because the MicroBlaze soft processor core is not a particularly good substitute for a DSP since it lacks special DSP features such as a MAC (multiply and accumulate) instruction, circular buffers, bit reversed addressing and saturating logic.
So a separate DSP soft core, such as the one described in this paper for the Xilinx Virtex-4, would be a better choice.
Many DSP designs would benefit from having both soft cores, since many if not most digital designs that include an FPGA also need a general microcontroller of some sort. As long as there are enough resources available in the FPGA (see below), soft processors such as the MicroBlaze not only eliminate a part in the BOM (and of course its associated cost), but also free up pins on the FPGA since there is no need to interconnect between the FPGA and a microcontroller. The space required for traces between the two parts is freed up as well.
The MicroBlaze can run at 210 MHz on a Virtex-5. Versions with an MMU can run Linux. A minimum MicroBlaze needs around 600 LUTs, and can grow up to 4000 if an FPU, MMU, cache, and other goodies are added. The DSP soft processor mentioned above used 1700 LUTs.
Since a Virtext-5 FPGA can have anywhere from 30,000 to over 200,000 LUTs, even including both of these soft cores represents only a fraction of the chip. Incorporating both allows for both conventional and DSP operations to take place in parallel, if desired, at the cost of some added complexity for synchronization between the two.
The IP for the MicroBlaze is free as long as you use it on a Xilinx FPGA and have licensed the ISE Design Suite Embedded Edition (or equivalent).
