Of Pipelines And Cybersecurity
One of the scariest phrases I have ever encountered is “gasoline pipeline” –thousands of miles of three foot diameter pipes buried in the ground for decades carrying billions of gallons of one of the most dangerous substances we encounter every day.
Reading about Colonial Pipeline shutting down its “network” on Friday raises many more questions than reporting addresses. If a headline says “cyberattack,” I am going to be confused when the article uses “network” to describe the actual pipeline, and not the routers and fiber that transport harmless packets. I noticed in previous reporting about last years “gas leak” in one of Colonial’s pipelines that additional confusion is introduced in the terms used to describe what is being transported in these pipelines. Sometimes they call them petroleum pipelines, sometimes gas pipelines.
To be concise, when Colonial says “gas pipeline” they are using the American English term for gasoline, not shorthand for “natural gas,” which, in reality, should be termed “gaseous methane.” I find that popular terms for noxious, combustible and explosive substances have been wordsmithed to appear less threatening than they are. Colonial also transports other fuels, like diesel, for home heating (heating oil) and transportation, as well as kerosene for aircraft.
Let me share a few stories about pipelines in general before talking about the Colonial news, and saying a few things about risk management.
The Energy Storage Pipeline Story
I heard this story second hand from a fellow engineer three decades ago. There was a large power company that built an energy storage facility in the south. It is very common to use gravity as a means of storing energy. During off-peak hours, the electricity output from a power plant is used to drive massive turbines to pump water uphill into a reservoir through giant pipes. During times of peak demand, the water is allowed to run back downhill through giant turbines to produce electricity. Sometimes, they are the same turbines.
All large infrastructure undergoes extensive computer modeling throughout the design phase. One step is dynamic analysis. Dynamics is the field of engineering that deals with vibrations, forcing functions, dampening and harmonics. If you have ever hit a piano key, you have experienced using a forcing function; the felt hammer hits a string and sets that string vibrating at its natural frequency. You may have used the soft pedal to dampen that vibration. The natural harmonic of a string is tuned by changing the tension on the string.
Everything has natural harmonics associated with it. The human stomach’s is two to three Hertz (cycles per second). If a car seat is designed improperly, it has a natural harmonic in the same range and will amplify those frequencies, which will make you carsick. (See this great video of the Tacoma Narrows bridge collapse to see one of the most famous examples of harmonics causing physical damage.)
Pipelines have natural harmonics. If the pumps or turbines are cycled at that frequency, they will excite that harmonic and the pipeline will vibrate. A very small, continuous excitation can be disastrous.
The engineer responsible for the dynamic analysis of the energy storage system carefully modeled the pipeline that led from the water intake to the reservoir. He determined the key dimensions like pipe diameter, thickness of its walls and support spacing to ensure that the operation of the turbines would not excite the pipeline’s natural harmonics. It worked, and the energy company moved on to its next project.
The next project was to build a similar storage facility in the West. This one was to be twice as big. To cut engineering costs, they simply doubled all the dimensions and added more turbo pumps. The engineer warned them that doubling all the dimensions while operating the turbines at the same speed would induce destructive vibrations. He was ignored, but proven right when the new design was fired up the first time. He reports that the huge pipes full of water, which were buried underground, jumped into the air and were completely destroyed.
The Ohio Pipeline Story
When I was at Gartner, I would take road trips to talk to customers about their IT security. I stopped in Cleveland a couple of years after 9/11 to talk to a major gas pipeline company. Until that day, I was unaware that buried in the ground all across the country are three foot diameter pipes transporting volatile substances; that “gas pipelines” were actually gasoline pipelines.
I sat in a small conference room with the company’s CIO and one of his IT directors. To start the conversation, I asked if their pumping stations were on a TCP/IP network. They did not know for sure but the director remembered seeing a network diagram circulated that showed routers and IP addresses for the pumping stations. So, yes.
Later, I asked my standard question: Do they have a problem with stolen laptops? No, was the CIO’s response. But the director knew of one case. The same guy who created that network diagram had his laptop stolen recently. Anything else? Well, a pumping station had been broken into, and the Cisco router had been stolen. You can guess what was going through my mind. Terrorists, sabotage, a networked gasoline pipeline, stolen laptop and router. Everything needed to create a disaster.
The Farewell Dossier Story
In the annals of stories about cyberattacks against pipelines, one stands above them all. It was first reported by Thomas Reed in his book At The Abyss: An Insider’s History of The Cold War, and then written up by William Safire for The New York Times. It starts with the greatest Cold War spy story ever, the Farewell Dossier, and ends with a bang, described by Guss Weiss in a PDF still available on the CIA’s website. In short, the U.S. planted back doors in the software of a Canadian pump manufacturer that was eventually deployed on a new Soviet pipeline. The pumps were “hacked” to excite natural frequencies which led to a massive explosion visible from space and was, at first, taken to be a nuclear explosion.
Keep in mind that this story has only one source, Guss Weiss. All references lead back to him. It may be a piece of CIA misdirection. Yet, its plausibility raises the specter of devastating attacks on pipelines.
Back to the Colonial Pipeline Story
Ellen Nakashima reports details in the Washington Post that make it seem likely that Colonial suffered a ransomware attack on its office systems and, out of an abundance of caution, shut down their pipeline until Mandiant could investigate.
So, what about risk management? Most organizations that treat cyber risk management as gospel attempt to identify all possible risks and then decrease the likelihood of a successful attack against high value assets based on those risks. They identify these key assets and put more effort into protecting them. But most of their assets are servers and databases, not physical systems. Some organizations–like pipeline operators–have to take into account risks like:
- A leak could cause hundreds of millions of dollars in damages to property and the environment.
- An explosion could kill people.
- A significant attack could put us out of business.
- If the attack causes us to shut down on the hottest or coldest day of the year, we could cause significant damage to the region we serve.
In other words, there are risks for fuel pipeline companies that could far exceed the value of the company.
What does it mean when a company with such high risks is compromised by ransomware? Should we believe that their pipeline control systems are completely air gapped from their corporate network? Are they likely to have invested in securing critical systems while leaving the rest of their systems vulnerable to common cybercriminals?
Shutting down a pipeline that feeds 45% of the Northeast’s fuel requirements is a big step. On one hand, it shows that Colonial understands the risks. On the other hand, it shows that Colonial Pipeline does not have 100% confidence in their operational systems’ cybersecurity defenses.
