When looking at different transformer core designs, three phased, I never see any core designed as a circle or torus.
Why is that so? Will it not work as well as the common B-shaped core?
Three coils, in magnetic series as you've drawn them, will not make a 3 phase transformer. There would be only one value for flux which would be common for all three coils, as each coil surrounds the entire core cross section.
In a real three phase transformer, each coil surrounds only part of the core, so that each coil can operate at a different flux.
A three-leg three phase transformer makes a saving in iron over three single phase transformers by sharing some or all of the iron return path.
To answer your comment on three-phase torroidal:
Because it seems, per wikipedia: Toroidal inductors and transformers, that the design should be superior. But I see no mention of three-phase usage, only single-phase.
Figure 1. 3-phase transformer flux. Source: NPTEL.
In a three-phase transformer each primary and secondary pair are wound on the same "limb" or "branch". With the 120° phase difference on each branch the flux on one branch can always find a path on the other two so that there is always a flux circuit. For example, when the red phase (Fig. 1) is max upwards the yellow and blue will be 0.5 downwards.
This arrangement is not possible on a standard torroidal transformer.
You could build a three phase transformer out of torriods. However, you need unique magnetic flux in each and the only way you could do that is to stack three separate torriods on top, or beside each other. Basically you would have three single phase transformers in one box.
I am willing to bet that historically 3-phase transformers were indeed built as three separate transformers till someone figured out that, since the three phases are 120 degrees apart, the magnetic effects of the other two coils basically cancel out at the primary coil in question. By combining them on a single core you can significantly reduce the weight and cost of the entire transformer.
In general torroidal transformers are expensive. Not only is the core itself harder to produce, but the act of winding it requires either very expensive knitting machinery or manual winding. That is an order of magnitude more cost compared to simple machine wound bobbins installed on laminated cores.
However, power toroidal xformers are made by winding very thin metal almost foil made by quenching very quickly so it has incredibly high permeability (I remember when this was new - I'm really old). I think it was first called Metglass? So in equipment to be shipped, if you care about weight, you might use toroidals. I have seen industrial higher powered equipment with three separate toroids used as three phase step down. I don't think it scales up to the power levels of "pole pigs" for utility distribution, and would probably not be cost effective.
You could use the shape of a wheel with three spokes, one primary and secondary winding on each spoke for each phase and no windings on the torodial wheel. But this is the same topology as the conventional three phase transformer with the B-shaped core described in the answer given by Transistor.
Will it not work as well as the common B-shaped core?
No, it won't.
Other answers already explained why a toroidal core is not suited for a compact three-phase-transformer. But even if that doesn't matter and you consider three single-phase-transformers, the toroidal core won't work in most applications involving three phases.
Toroidal cores work well for instrument transformers, transducers and other applications where no significant power flow occurs.
Three-Phase-Transformers are almost exclusively used for high-power applications, e.g. to connect generators and motors with the electrical grid and to transform voltages within the grid. In any case a high amount of energy is involved. To transport this energy you actually need leakage fluxes, which you (almost) won't have in case of a torodial core.
If you load a torodial transformer with a high current, the secondary voltage will be reduced significantly or even vanish.
The whole thing is not easily to understand and let to a lot of discussions under my colleages. To get some deeper insight I'd recommend you some literature to start with:
Edwards, J. and Saha, T. K. (2000). Power flow in transformers via the poynting vector. In: A. Krivda, Proceedings of the Australasian Universities Power Engineering Conference: AUPEC 2000. AUPEC 2000, Brisbane, Australia, (86-91). 24-27 September 2000.