The world’s most important facilities—think massive hydroelectric dams and nuclear power plants—are vulnerable to devastating cyberattacks. And it may be just a matter of time before someone gets hurt.
But nobody panic.
That’s the overwhelming takeaway from new research set to be unveiled at the Black Hat cybersecurity conference in Las Vegas next week. The researchers have already gained the attention of major industries, but it remains unclear whether they will fix the problem before it’s too late.
The trouble centers around vulnerabilities in so-called Industrial Ethernet Switches (IES), the devices that create the internal networks that are vital for the function of modern factories, refineries, ports, and countless other industrial environments today. The critical vulnerabilities in IES allow attackers to gain access to the network, take full control, and cause potentially fatal damage, the researchers say.
“There is a massive lack of security awareness in the industrial control systems community.”
Industrial switches are ubiquitous in today’s networked industry but rarely appear in homes, making them unfamiliar for most people. But the instrumental role they play in countless facilities means any single vulnerability has far-reaching consequences.
The vulnerabilities can lead to events reminiscent of the 2010 Stuxnet attack on Iranian nuclear facilities or the 2014 cyberattack on a German steel mill. These attacks were the first time purely digital weapons caused physical damage to their targets. Stuxnet shut down a wide swath of Iran’s nuclear facilities, while the 2014 attack caused “massive” damage in the German facilities when the factory owners were unable to shut down a blast furnace.
“Anything that the facility is capable of in its natural operating system, you’re [an attacker] capable of doing—and doing damage with if you control the network,” Robert Lee, a security researcher and and active-duty U.S. Air Force Cyber Warfare Operations Officer told the Daily Dot.
“With a power station, you can have major repercussions. With a hydroelectric dam, if you don’t monitor processes in a normal situation, it’ll spin out of control. Everything you have can be manipulated.”
Funded by security firm IOActive, the research came out of a collaboration between Lee, risk researcher Eireann Leverett, and IOActive security consultant Colin Cassidy, who sought to find and study IES vulnerabilities. The group has been working with at least four industrial switch vendors—Siemens, General Electric, Opengear, and Garrettcom—to disclose and fix, as best they can, the exploits in question. Except for Opengear, the vendors did not yet respond to a request for comment.
While these companies are working to fix the problem, the actual process of patching the switches can take several years and piles of money to accomplish, leaving large numbers of industrial facilities open to attacks on their network today. A major part of the researchers’ work has been developing mitigation techniques to defend against IES attackers even before a patch is implemented.
The vulnerabilities on industrial switches covered in the new research include the widespread use of default passwords, hard-coded encryption keys, and a lack of proper authentication for firmware updates. These three fundamental failures of security combine to make it easier for attackers to gain access to industry devices and networks, change what they please, and take control.
In some cases, remote attackers are able to download all the diagnostics and configurations to which only administrators should have access.
Forged session IDs, cross-site scripting, and cross-site requests allow a hacker to do things like surreptitiously create a brand administrator user account on the switch as a backdoor to control the network.
Backdoors also exist in the form of hidden accounts originally created for maintenance that can provide cover for attackers. In particularly insecure facilities, antiquated and unencrypted connections to the Internet that allow engineers remote access to their networks act as pathways an attacker anywhere in the world can take toward the network in her crosshairs.
“All these vulnerabilities are pervasive and endemic,” Leverett explained. “Most vendors haven’t done the basics.”
The vendors themselves explain the problems as a result of outdated thinking and technology.
“A lot of it comes down to the mindset when this [equipment] was installed four or eight years ago,” Robert Waldie, an Opengear engineer who continues to liaison with the researchers, said in a phone interview. “There was a lot less emphasis on hardening and ongoing auditing on this kind of equipment. They were just part of the background infrastructure. But now there’s a realization that these are significant entry points into critical infrastructure. Anecdotally, I think there’s a lot of vulnerable equipment out there.”
“With a hydroelectric dam, if you don’t monitor processes in a normal situation, it’ll spin out of control. Everything you have can be manipulated.”
Moreover, these vulnerabilities are not an exhaustive list.
“Everything they look for, they find,” Lee said, describing the research efforts of his colleagues. “The specific vulnerabilities get attention; but the fact is, whenever Eireann and Colin look for something, they find it. It doesn’t take them as long as it should to find these things. A lot of these are low hanging fruit that have huge impacts.”
Incredibly, it can take up to three years to fully fix any given problem. The process is slow and costly.
First, the researchers notified the federal government or another third party of the vulnerability and then the vendors themselves. Then, it took switch vendors like General Electric eight months and Siemens three months to issue patches that sometimes only fix a portion of the problem. While the patch then exists, research shows that industrial facilities don’t actually implement the patches for up to 18 months afterwards.
Implementing a patch on critical hardware like the switch involves jumping through numerous hoops with management and then bringing the entire network down, which can cost thousands or millions of dollars every hour, according to the researchers. For that reason, many of these facilities are almost never patching more than once a year, and the timespan is often much longer.
“It’s been a mixed jar in terms of speed,” Waldie said. “A lot of industrial systems are very slow moving and, paradoxically, the data center is quite a slow moving place in terms of adoption for new software and things like that, especially for embedded-type systems. What we’ve found is the best approach is to disclose as quickly as possible for severe issues and to include mitigation instructions.”
But even before a patch, there’s plenty to do to defend against attacks.
“In some cases, mitigations are in place if it’s a configuration item that’s not correctly set,” Colin Cassidy, a member of the research team, told the Daily Dot. “The most secure deployment [of switches] isn’t always the default configuration and vendors are switching.”
Changing default configuration choices means giving up some backwards compatibility, a difficult choice made easier as research reveals how potent a cyberattack can be.
Industry system administrators must “look at all the user accounts on switches and disable the ones not being used,” Cassidy explained. They must also “swap out all the hard-coded encryption keys, change default passwords.”
Network monitoring is a huge blind spot, according to the researchers. Standard intrusion detection systems keyed up to the facility’s user accounts and maintenance can alert a defender whenever, for instance, an unauthorized account is uploading hacked firmware to the switch in order to take control.
“Updating firmware without authentication usually originates from a computer that shouldn’t be accessing the switch anyway in the middle of the day, plus a massive flow of data that shouldn’t be there either,” Lee said. “Monitoring can stop that.”
But to really fix the problem, the researchers insist, a culture change is required.
“People look for magic bullets—and there are discussions to be had, like [ones about] smarter network monitoring—but with so many things, we already know how to fix them, we already have the settings,” Lee said. “What we don’t have is awareness. There is a massive lack of security awareness in the industrial control systems community.”
The presentation on these vulnerabilities is expected to grab such a big spotlight at Black Hat that a fully fledged press conference is being planned around it.
The trick is that, while most hackers are worried about gaining administrative privileges on their targets, almost anyone on an industrial control system network already has such privileges. Often, merely gaining access puts an attacker in a position to do damage, the researchers say. That’s why insecure connections to the Internet by switches, for instance, were described as so “unfortunate” by researchers.
These devices can be found all over: In electrical facilities, food processing plants, manufacturing plants, on board ships, transportation facilities, and more. Even the researchers are surprised with how ubiquitous these devices are. They described walking into facilities and, on multiple occasions, discovering vulnerable network switches they didn’t expect to find.
The threats to nuclear power plant security are among the most attention grabbing. The researchers have done work in nuclear facilities and described the state of the industry’s security as serious but often over-confident.
“The downside is that, because of some very strict policies, they generally sense that their security is better than it actually is and they don’t have to worry about these issues,” Lee said. “That’s simply not true.”
While the facilities themselves are big on security measures that directly protect them, the researchers often found paths from the enterprise-side of the business into the industrial networks.
“If we look at the type of adversary that would use these vulnerabilities, these are nation-states,” Lee said. “The motive isn’t there for crimeware and hacktivists. If you’re looking to control the entirety of a network for massive espionage or prepare for actual damage, if you’re looking at who targets nuclear centers, you’re generally one of the big players: Russia, China, Iran. Those targets are very interesting to you, to have access to an adversary’s center of gravity.”
The Chinas, Russias, and Americas of the world invest immense resources in entering the networks—like, for instance, Stuxnet.
“The getting in is some other research, but what they do while they’re in there are these vulnerabilities,” Lee explained.
The game isn’t over once an attacker is inside a target’s network. It takes a lot of effort to remotely and stealthily map out the intricacies of an industrial network that are often uniquely designed. The first step to all of that is to gain network access through the switch.
“All these vulnerabilities are pervasive and endemic. Most vendors haven’t done the basics.”
The fact that it’s so tricky to operate in an industrial network means more-familiar defenders have an advantage over attackers, one that ought to be multiplied with smart monitoring of data, the researchers said.
When Stuxnet was used to attack Iran, one tactic was to disable the industrial monitoring systems so that when sabotage occurred, the computers couldn’t alert anyone. From the inside, everything looked normal. The researchers repeatedly pointed out that gaining control of a network can yield that kind of result.
“Look at what happened in the San Bruno pipeline explosion,” Lee said, referring to a 2010 industrial disaster that killed eight people in California.
“When you look at the normal operating conditions in control systems like a pipeline, you can reach situations that will cause the loss of human life. Without even trying to do anything malicious to the process itself, an attacker can be working while alarms are coming back saying that we’re about to reach max pressure, and those packets can be dropped, that traffic denied. Not only can it lead to the loss of human life and physical damage, it can lead to immense difficulties in sending emergency response going to the right location because the monitor’s data is compromised.”
In the midst of announcing critical vulnerabilities, the researchers say they are wary of the hysteria that too often takes hold in media about cybersecurity news of this magnitude. While the consequences of these vulnerabilities are very big, they’re not without limits. A cascading power grid failure, a regularly-featured nightmare in cyberwar news, is outside of the realm of possibility here.
Lee has devoted significant work to debunking hysteria and myths around cyberwar. Earlier this year, when news of Iran’s supposed cyberwar against the United States hit the New York Times, Lee made a detailed argument about why those claims were greatly exaggerated.
“We need less fear in the [security] community,” Lee said. “I’m concerned about people saying, ‘This is how Iran takes down the power grid.’ That’s just not the case.”
Update 12:54pm CT, July 29: Added details about security firm IOActive’s role in conducting the research.
Photo via J Brew/Flickr (CC BY 2.0) | Remix by Jason Reed