In the aftermath of Winter Storm Uri, Texans spent a week in darkness, leading to dozens of deaths from freezing cold, hundreds of carbon monoxide poisonings, and multiple second-order effects like water plant failures and communication outages.
News headlines have talked about “how Texas was only 4 minutes and 37 seconds away from total grid collapse with devastating consequences.”
Texas was 4 minutes and 37 seconds from a collapse of the power grid. pic.twitter.com/K3rXtEx7Tf
— Matt Largey (real) (@mattlargey) February 24, 2021
There have been a handful of attempts to explain how things got to that point, but we haven’t seen a good explainer of just what ‘total collapse’ means in this context and why that would have resulted in weeks or months of major pain.
Why wouldn’t it have been as simple as fixing the specific broken lines and pipes then getting back to normal?
Electricity can be a finicky thing. So the most basic takeaway from this is to understand the grid is not run with a simple on-off switch — instead, picture a street performer trying to spin lots of plates at once, in perfect synchronicity, with one arm tied behind their back.
https://gfycat.com/distantunpleasanthyracotherium-spinning-plates-overwhelm
Summary:
- The problem is electricity frequency, not the voltage, amperage, etc.
- Similar to a choir singing in harmony, grid operators care about keeping frequencies just right (typically 60 Hz) and in harmony across the different pieces of the grid.
- The key is balance. Grid operators work around the clock to keep the grid frequency in harmony. But it’s a constant cat-and-mouse game because demand and supply fluctuations are always influencing the frequency.
- When frequencies get out of whack, that can cause all kinds of cascading problems that aren’t easy to fix, such as burning out engines and other critical components.
- Part of why things can snowball into collapse so quickly is that a particular plant or piece of equipment can have protective cut-off relays that automatically isolate themselves from the grid to prevent those grid frequency problems causing local hardware damage.
- But those cutoffs happen at the same time the grid needs more capacity plugged in, not less, so it accelerates the death spiral.
- Since the Texas grid is isolated from the rest of the United States, it couldn’t easily import electricity from other states to maintain a stable grid frequency.
- During the Uri storm, two things happened: supply went down as lines, pipes, and other equipment failed due to the extreme weather, while at the same time demand spiked as people tried to heat their homes.
- That caused the grid-wide frequency to start dropping. So plant operators had to quickly shut down certain customers (to lower demand) and bring things back in harmony.
- If they had been just 5 minutes slower (which was around 2 AM, no less), all of the spinning plates would’ve come crashing down.
- And that’s where the weeks-months of aftermath would come in: once the whole grid goes down, it’s difficult and slow to boot it back up again due to the ‘black start’ problem. It’s just not as simple as ‘turning things back on.’
- Put simply: going from 0% to 1% is harder than going from 50% to 100%.
What makes the Texas grid unique
North America’s power grid is split into two main grids: the Eastern and Westerm Interconnection. Most of Texas is not on either of these grids but is instead on what’s called the Texas Interconnection, which is managed by the Electric Reliability Council of Texas (ERCOT).
The good news for Texans is that if the rest of the country goes down, the Texas Interconnection might stay up. But that obviously can be a problem when Texas needs help and can’t get it from other states. Texas does, however, have connections to Mexico and sometimes buys electricity from our Southern neighbor.
What’s the frequency, Kenneth?
The power you get from an outlet is alternating current (AC), called so because it alternates between positive and negative polarity several times every second. That’s called its frequency, which is measured in hertz (Hz). In the image below you can see how AC (in green) moves between positive and negative in a waveform.
Different countries have different standards for utility frequency. In the United States, it’s 60 Hz. In most of Europe it’s 50 Hz. But things weren’t always so standard.
The early days of electrification were messy and it took decades to establish standards. Until 1948, the Los Angeles power grid operated at 50 Hz while the rest of the country was at 60 Hz. Manufacturers had to make special versions of electrical products just for the Los Angeles area to account for the frequency difference.
Why does frequency standardization matter? Because if you plug a device designed to operate at 60 Hz into an outlet that provides the same voltage and amperage at 50 Hz, it won’t work correctly, especially anything with a motor. Clocks would lose time and record players would play slowly. That especially affects things like, say, the delicate equipment in a power plant.
Frequency incompatibility is still an issue in Japan today. Half of its grid operates at 60 Hz while the other half operates at 50 Hz. That was a big problem during the Fukushima meltdown, because the 60 Hz half of the country couldn’t help the other half when it needed more power.
We spoke with Robert Zahm, a retired electrical engineer with 37 years of experience, 26 of those working for Niagara Mohawk Power Corporation, for additional understanding of how this all works. He explained that power plants are composed of multiple generators, engines in other words, that all have to run in perfect synchronization. Zahm explained that there are ways to bridge unequal grids, but that need has to be known ahead of time and it takes large and special systems to do so.
Let’s say you have two engines working together, each at slightly different speeds. The slower engine tries to slow down the faster engine, while the faster engine tries to speed up the slower engine. This puts incredible stress on the couplings, which could cause them to break. It’s just as if you were driving at 80 mph in your car and suddenly shifted to a lower gear: the car would come to a sudden stop and you might break your engine, transmission, and the couplings in between.
Regardless of the frequency, it’s very important for all players in the power grid to stick to it. An unanticipated deviation of just a few kilohertz (kHz) could destroy vital equipment. And it takes a careful balancing act to pull that off all day every day.
Load balancing the grid
To maintain a steady 60 Hz frequency, the grid must be perfectly balanced: electricity must be consumed as it’s generated. What goes in must go out. If there is more electricity in the grid than is being used, the frequency increases. If there isn’t enough electricity to meet demand, the frequency drops.
Much of the work of grid operators is keeping the grid in balance, which is called load balancing. That’s also why renewables like solar and wind are problematic within the context of these older systems: if the wind gusts or stops, or if clouds shade the sun, it can throw the grid out of whack and operators have to scramble to compensate.
(Though traditional methods of generation aren’t perfect either, and this in no way is an argument against moving to renewables as a society.)
In a large interconnected grid, smaller grids can trade power to balance each other. In a grid like the Texas Interconnection, that isn’t an option.
When multiple power plants in Texas went offline and demand skyrocketed, generating more power to account for the demand wasn’t an option, so the only choice on the table was load shedding. In other words, rolling blackouts to reduce strain on the grid and maintain a steady 60 Hz frequency.
If the grid gets off frequency, it could damage power plants, so they are protected by relays that disconnect power plants from the grid before that happens. They work similarly to breakers in your house that cut off power to circuits before your wiring and appliances are damaged.
On Sunday, February 14th, power plants were being knocked offline while demand surged. The real kicker was when the South Texas Nuclear Generating Station was knocked offline because a five-foot section of uninsulated pipe froze, which cut capacity by 1,350 megawatts — enough to power 270,000 homes.
In the wee hours of Monday, February 15th, the manure was starting to hit the fan. At 1:51 AM, the grid dropped below 59.4 Hz. If it stayed under 59.4 Hz for nine minutes, it would have triggered the relays, causing a cascading failure that would have knocked most of the power plants in Texas offline.
This went on for four minutes and 23 seconds as the frequency fell to 59.3 Hz. Grid operators had already been implementing blackouts to shed load, but it wasn’t enough. Thankfully, they cut off enough homes from the grid to stabilize the system by 2:03 AM.
Bill Magness, president and CEO of ERCOT, later said this about the situation:
That is the thing that we cannot allow to happen. Because if we have a blackout of the system, the system is out for an indeterminate amount of time. We may still be here today talking about ‘when is the power going to come back on’ if we had let the system get into that condition.
Once an entire grid is offline, it’s extremely difficult to jump start. It’s much easier to go from 1% to 100% than from 0% to 1%.
Black start, black swan
As the old saying goes, it takes money to make money. That also applies to electricity: it takes power to make power. Power plants run off electricity, and if they’re not generating their own power, they have to pull from another plant. If there isn’t another plant to draw from, that’s a big problem.
If the Texas grid collapsed, ERCOT would have had to pull off a difficult and laborious process called a “black start,” in which the entire grid would have had to be bootstrapped back into working order.
Black starts are extremely difficult to pull off, because again, the load has to be perfectly balanced. It’s not enough to fire up a diesel generator and boot up the plant — there has to be an evenly matched load on the line as well. Additionally, damage could have occured in the system that couldn’t have been identified until power had returned.
And if things get out of balance during the black start, more equipment could be damaged, leading to even more setbacks. Zahm said that utilities essentially have to rebuild the grid a piece at a time, which is part of why it takes so long.
The easiest type of power plant to black start is hydroelectric, because it takes relatively little power to get back online, and hydroelectric plants are more tolerant of frequency fluctuations. The most difficult plants to black start are coal, nuclear, and renewables.
Zahm explained that the reason why coal and nuclear are difficult to restart is simple physics. Coal and nuclear both work off the same principle: heat water to boil it and spin power-generating turbines. It takes time to build up that level of heat, as much as a couple of days.
Unfortunately, Texas has very little in the way of hydroelectric generation, making up about 1% of its power supply. And since Texas is cut off from the Eastern and Western grids, it couldn’t pull power from the Hoover Dam or the Tennessee Valley Authority. It would have been a herculean effort to slowly bring the natural gas and nuclear plants back online.
However, Greg Piesco-Putnam, CEO of Cosine Energy, told us that he doubts it would have taken that long to get the Texas grid back online:
Is it possible that it could have taken weeks to restart the Texas grid, sure. But it would have required black swan events on black swan events. For example, if the Polar Vortex continued to sit over Texas for several more weeks, it would have been very difficult to get the grid back to normal. However, even in that world, the grid would be operating at 80%.
But even then, there would have been some hiccups:
A gas plant can go from cold to full power in about 90 minutes. But a nuclear plant will take between 12-72 hours to power up. Given that nuclear provides approximately 12.5% of Texas’s winter power, that would represent 4M people being stuck in rolling blackouts until they were restored.
Piesco-Putnam also said things would have been rougher in rural areas. “You could imagine a situation where enough transformers were damaged that isolated rural houses could be without power for a few weeks as linework would need to be done. But that would be a low single-digit percentage of the grid customers,” he said.
While black starts are difficult to pull off, they have been successfully pulled off in the United States multiple times, such as in the 2003 Northeast blackout, in the aftermath of Hurricane Katrina, and in Puerto Rico after Hurricane Maria.
“In the 1965 Northeast Blackout, the black start began within 8 minutes of the blackout and power was restored within 13 hours. In the 2003 Northeast Blackout, most people had their power restored within seven hours. So historical [comparisons] suggest these types of events are far more likely to resolve in hours rather than days, weeks or months,” Piesco-Putnam said.
Zahm told me that a big part of why the 2003 blackout was resolved quickly is that they managed to keep one of the coal plants burning, which made it a lot easier to black start than if it had been cold.
However, just because most power outages don’t last that long, that doesn’t mean you shouldn’t be prepared. “It is totally reasonable for someone to be prepared to survive a week in really rough weather conditions without access to external resources, be it water, power and gas,” Piesco-Putnam said.
And Texas particularly seems to have a hard time with black starts. A report on the 2011 winter storm said, “Many of the generators that experienced outages in 1989 failed again in 2011.” So even after 22 years, nothing much had improved:
ERCOT has 15 primary and six alternate black start generators. During the event, roughly half of these generators were unavailable: two (totaling 97 MW of capacity) were on planned outage; four (totaling 141 MW) failed to start due to the extreme cold weather; three (totaling 423 MW) tripped offline after starting due to freezing equipment; and one (26 MW) tripped offline due to natural gas fuel curtailment. Had a total blackout of the ERCOT system occurred, the unavailability of 10 of ERCOT’s 21 (primary and alternate) black start resources, comprising 687 MW out of a total 1150 MW of black start capacity, could have jeopardized ERCOT’s ability to promptly restore the system.
The big question is if ERCOT will finally learn from its mistakes.
Don’t expect these fundamental grid problems to go away anytime soon
Zahm explained that he expects the power grid to become more unstable over time due to two factors: deregulation and increased reliance on unpredictable renewables like solar and wind. But Piesco-Putnam is quick to reply that the real underlying problem behind those two surface factors is the government’s and ESCOTs failures to prepare.
Why is deregulation a factor? Capitalism often means that companies are squeezing out every ounce of efficiency they can. That means they don’t like waste or “slack” in the system because it hurts their bottom line.
But that slack is what can help smooth out black swan events. No slack = no safety net. “The deregulated system is strangely efficient, but at the expense of redundancy,” Zahm said.
Some grids outside of Texas have intentionally decided to keep/build in some slack. Texas didn’t. Combine that with a general failure to prepare — eg. inefficient black start equipment, uninsulated pipes, etc. — and you’re simply gambling that things will work out.
Some people were quick to blame the Texas failure on renewable energy, but we suspect it was due to political reasons or simple greed.
There are valid, practical challenges that come from plugging renewable generators into the old grid. Again, power has to be used as it’s generated to maintain a stable grid frequency, and with solar and wind you’re completely at the mercy of nature. Our power grid has to run with the precision of a Swiss clock, and nature just doesn’t work that way.
But the problem isn’t the idea/trend of renewables, it’s the lack of effort to fix “the flip side of that coin.”
For example, Zahm explained that batteries will have to be integrated into the grid to store reserves of renewable power, something the Biden Administration is working on, but batteries have their own problems: they’re full of toxic chemicals, only last a few years before being exhausted, and many experts doubt that the technology is there yet to store enough energy to be truly useful on the grid.
“We’re heading further and further down this path. That’s why I put a generator on our house,” Zahm said.
Don’t be left in the dark! Check out our guide to the best portable generators.
You are reporting the comment """ by on