MarkWest Energy Partners, L.P. is a recently acquired subsidiary of Marathon Petroleum that gathers, processes and transports natural gas from the Utica Shale region in Ohio. Four facilities about 90 minutes apart each have DeltaV systems, and needed to be brought into compliance with Marathon standards for cybersecurity.
“Marathon has a pretty good expectation on the IT side,” said Paul Miano, cybersecurity operations, Marathon Petroleum Co. “We want to bring the lessons we learned on the IT side to the OT side and align them, with the adjustments needed, to meet business objectives.”
MarkWest lacked the resources to manually patch and backup systems in accordance with the policies and wanted to make a comprehensive plan for improvement. “Cybersecurity can be overwhelming,” Miano said. “A cafeteria tray approach doesn’t work!”
Miano, with co-presenters Ron Sprengeler (pictured), operations project lead, MarkWest Energy Partners, and Ben Jackman, lifecycle services program manager, Emerson Automation Systems, presented “Exemplary cybersecurity improvements: Applying DHS Seven Strategies to defend industrial control systems (ICS)” at the 2017 Emerson Global Users Exchange this week in Minneapolis.
Read the paper
The Marathon/MarkWest approach is guided by the U.S. Department of Homeland Security (DHS) white paper, “Seven Steps to Effectively Defend Industrial Control Systems.” “It’s a six-page paper that anyone can read and understand,” said Jackman. “It applies to all industrial control systems, and gives a good background for getting started in cybersecurity.” The paper describes seven steps in descending order of the percentage of reported industrial cybersecurity incidents they would prevent:
“These seven strategies provide a defense-in-depth environment – the layers of protection in the onion model,” Jackson said. The presenters related the steps to actions in their industrial facilities:
Implement application whitelisting: This can help detect and prevent attempted execution of malware. “Any application you don’t list is not allowed to run,” said Miano. “This limits what will be in the soft, gooey middle of the onion.” A control system is reasonably static unless you’re doing updates, and certainly can be addressed with whitelisting. Miano said, “Do calibrate your list to be sure that what you allow will run without what you leave out.”
Ensure proper integration and patch management: “Any good patch management system begins with a complete inventory and systems baseline,” said Sprengeler. “We thought we knew what our hardware inventory was, but until you get down and dirty and into it, you don’t, PLCs and all.” With software, some things that look like you need, you don’t, “such as HAVEX,” Spengeler said. HAVEX is a form of malware—a remote access trojan (RAT). “People are getting tricky about getting their payloads into your environment,” he said.
Validate downloaded software with digital signatures and vendor-supplied hashes. Pay attention to PLC/RTU firmware and the required configuration software, and limit or eliminate connection of external laptops to the control network.
Reduce your attack surface: This means isolating your ICS networks, especially from the internet. Allow connection only when there is a defined business requirement: historian, supply management, etc. “Make sure your networks are locked down, shut down unnecessary ports, and allow no additional software, like MS Office, where it’s not needed,” Miano said. “That also limits the patches you have to do.”
On the question of when to use a firewall versus a data diode (a one-way connection where data only flows out, not back into a system), Miano said, “At some point, data transfer is inevitable, for patches if nothing else. You have to manage that. Use a firewall anytime you need access from the outside.”
Build a defendable environment: Begin with a solid physical perimeter, and use identity verification systems. “Have card access to your server rooms, and key access to computers,” Sprengeler said. Network segmentation limits damage from network perimeter breach or bypass of physical security, and prevents widespread effects if malware is introduced. “If someone gets on your layer 2.5 network, can they go everywhere? Limit that,” he said.
Restrict host-to-host communications. “Don’t let one server talk to another unless you have to. Then firewall it,” Spengeler said. And have effective removeable media methodology and procedures. “There are always arguments to allow some removeable media, at least for emergencies, so you have to have a policy and limit it,” he said.
Manage your authentication: “Restrict the who, what and why of what an individual can do,” said Miano. “Allow no unauthorized individuals.” Limit admin access to those who need it, and separate ICS and general admin roles. Lock up high-level accounts so they are not exposed if someone gets in. “Don’t let the VLAN and ICS touch,” he said. “You don’t want a hacker on your network to be able to get into DeltaV.”
Emerson’s Jackman added, “Combine something you know, like a password, with something you have, like a key.”
Monitor and respond: Monitor the network for suspicious activity. “It’s easy to be able to see suspicious activity with an intrusion detection system,” Spengeler said. Baseline the network traffic to learn your “normal,” and implement security information and event monitoring (SIEM) to track logins, and monitor use and misuse of administrative privileges. Manage vulnerabilities using manual or automatic scanning.
Finally, Jackman, said, “Have a response plan in place, and a way to recover.”
Implement secure remote access: “Just because it’s the last 1% doesn’t mean you can write it off,” Jackman said. “Some of these events are a lot more severe than the others.”
Use a read-only data diode for monitor-only access. “Look at a data diode. Can it provide the data you need out of the plant?” Miano said. Allow no persistent connections for vendors or business users, he added. “Our preferred methodology is to use an operator-controlled connection, an operator switch that can be used to let an outside user in when needed, then shut them back out. Think lock-out, tag-out procedure.”
Wherever you can, use two-factor authentication. “The attack that shut down the grid in Ukraine was a brute force attack on an account. Two-factor authentication would have prevented that,” Miano said.
Sifting it into DeltaV
Emerson offers a range of products to help support the seven steps in DeltaV installations. “We took a look at all seven and implemented pieces of them,” Miano said. They include, for each step:
1. Application Whitelisting for DeltaV Systems
2. Automated Patch Management
3. Emerson Smart Firewall, Workstation Hardening
4. Firewall IPD, Tofino Firewall, USB Protection
5. Multifactor authentication, DeltaV User Manager, Active Directory
6. SIEM, Security Monitor, Backup & Recovery
7. Multifactor authentication and Remote Desktop Gateway
Sprengeler said, “In some cases, we worked with Emerson, and in others, we used established Marathon corporate strategies.” In the process, MarkWest was put on the path to improved consistency and efficiency of cybersecurity practices.
It began with a security baseline and implemented a consistent methodology to meet, exceed and report against the baseline. Where possible, it is eliminating human effort by automated patching and backups. “The manual effort to identify and apply security patches and implement consistent, reliable backups is high,” said Jackman. “Almost a full-time resource is required to address MarkWest’s four-plus DeltaV systems and more than 50 workstations. Automated tools have helped alleviate that manual effort and eliminate human error.”
The DHS Seven Steps provide a vendor-agnostic framework and reference for selecting solutions, and MarkWest was able to apply many of the steps with selection of a few of Emerson’s cybersecurity solutions for DeltaV systems.
In the end, Miano said, “The primary thing we’re trying to achieve is system reliability.”