Note that they very specifically meant what they said - often statements like that have a degree of meaningless gobbledygook in them, but their explanation makes sense.
- “Power sources must be phase-synchronized before they can be brought online to load. Bringing these generators online required manual synchronization. We’ve now restored power to the Availability Zone and are bringing EC2 instances up.”
From information provided one cannot of course be 100% sure of what happened but it could have gone like this, and if not exactly this then what happened was probably a first cousin of this.
Initial problem causes fire and explosion and destroys power equipment.
Breakers open, load is transferred to new places, inductive surges from transformers suddenly having power taken off occur, voltages on circuits fed by the same upstream transformer fluctuate.
Somewhere somebody hasn't done their homework well enough. The circuits feeding the transformer that power the center also power a circuit into the alternator plant. Not the same circuit - a notionally unrelated upstream one. Not even the whole power feed to the alternator area. Maybe just one phase. Maybe only part of the areas total feed. But ...
Upstream of the incinerating transformer the "transient load" is too much. May be phase imbalance, may just be a hardish short on the transformer primary. May be ... (Murphy has fun) ... . And the upstream breaker trips as well.
In the alternator shed its all go, contactors operate, diesel starters cough them into life, alternators spin up, system steadies at 400 vAC (probably). Ready ...
Now to synchronise these babies. I said synchronise. Synchronise. Hello ...? In the synchronisation unit there is no power. Somebody has messed up. The tripped breaker or some similar fault has touched just the periphery of the standby area. The alternators are running but, because there are several of them they need the conductor to link them together to get enough combined power to bring up the data centre load safely. Or some such similar scenario. But the sync gear is dead.
When brains realise what has happened women (or men) are sent to do the job manually. Somebody get's to do real live phase syncing like they taught you back in tech/university/ wherever and you haven't done for years. Books are looked at. People nervously implement what they haven't practiced for too long. Hooray! It works. Amazon EC2 is back on line.
Somewhere somebody is about to lose their job !!!
Where you have two AC mains power sources that are electrically connected they MUST both run at the same frequency and be very close to being synchronised in phase. One source may be the mains "grid" and the other may be an alternator or more than one alternator. In this case, based on their statements, a number of alternators were involved. It is likely that in order to startup the EC2 centre an amount of power was required that was in excess of what could be provided by a single generator. That means that in order to start at all several generators needed to first be running as if they were a single unit - ie, in synchronisation. Without the ability to synchronise, the generators could all be running but be "unable to pull together as a team".
Once in synchronisation if a source increases in phase wrt the others it delivers more power and if it starts to lag in phase it delivers less power and as this is increased it becomes a 3 phase motor being driven by the mains.
At startup when the sources are separate and must be connected, synchronisation is not hard but sources must be tracking accurately, identical in frequency and accurate in phase to within a small fraction of a cycle. There are ways of doing this with nothing than a few lightbulbs and some wires and a heaping helping of bravery, for larger systems, which tend to go bang if you get it wrong. BUT in emergency standby systems the synchronisation is carried out by automatic equipment which controls the alternator's diesel engine speed and then nudges the alternator into synchronisation with the other source by accelerating or braking it as required. This is not a hard job to do BUT if the equipment fails to work (as here) then the alternators can be working perfectly but not be able to be used.
This is a common industry term.
"Upstream" refers to a portion of the circuit that is closer to the source of supply than the current point of reference.
So an upstream transformer is one which is closer to the power station or local switch yard.
As you progress "upstream" from the point of use and encounter transformers it is usual that they will be progressively larger and have greater fault handling capability. The design engineer attempts to dimension the upstream equipment such that it's fault protection equipment is NOT operated when the protection for a further downstream unit is operated. In this case, a fault at the data center would have not been intended to trip the protection equipment at a transformer some distance away which fed supply to the location. It would have been supposed that this remote transformer would not be affected by a local fault which required a motor alternator startup.
However, only so much fault protection can be provided economically and in this case something happened which exceeded design expectations and the fault "propagated". We cannot tell whether it propagated upstream to a shared common source as in my explanation above or "sideways" where power from one system eg caused arcing which damaged another adjacent circuit. We do know that it did happen (they tell us).
Several examples of upstream transformers and fault issues.
Example of sizing an upstream transformer to handle SMPS fault capability Get a bigger fault than xpected and ...
Accident involving upstream and downstream transformers][(http://www.eea.co.nz/Attachment?Action=View&Attachment_id=849) Not identical but demonstrates terminology and relationship
Effect of load transients on upstream transformers - in this case SCR switching transients.
Motor and transformer sizing example. Get it wrong or have an exceptional fault and ...
Q: How to size transformer for dedicated three phase motor loads?
A: For easier and quicker estimation, someone might prefer to calculate motor starting current as minimum transformer size. This, however, is overestimation. The correct approach is to calculate transformer short circuit capacity bounded by upstream transformer and see if your chosen transformer match up transient starting current of motor loads. You should consider also how many motors started within an hour