The technologies behind industrial control systems (ICS) drive the core operational processes in industries such as manufacturing and critical infrastructure. Many organizations in these sectors offer essential services that communities rely on, and they are often dependent on ICS technologies to monitor and control critical processes. Adverse effects on these technologies could have detrimental consequences, which is why securing ICS environments is a rising priority and should be regarded as a top concern for security teams managing operational technology environments.

The Industrial Control Systems Security Landscape

X-Force Red is an autonomous team of veteran hackers within IBM Security, hired to break into organizations and uncover risky vulnerabilities that adversarial actors may use for their own gain. X-Force Red offers penetration testing and vulnerability management programs to help security leaders identify and remediate security flaws covering their digital and physical ecosystems.

According to X-Force Red data collected from our vulnerability database, the number of vulnerabilities exposing industrial control systems has increased 83 percent since 2011. This doesn’t only mean that these systems are likely becoming more vulnerable every year; it may also mean that already vulnerable systems are possibly under attack, which would expose issues that were there before and now come to light. Furthermore, the rising awareness of threats to ICS-rich environments is resulting in better documentation of vulnerabilities and flaws.

Seeing the number of vulnerabilities rise over time is evident in X-Force Red’s ICS penetration testing results. When we test legacy ICS environments, we discover many severe vulnerabilities, some of which may have been exposing the system to potential attacks for years and could be easily exploited by an attacker.

Figure 1: ICS vulnerabilities discovered annually (Source: X-Force Red)

What drives the continuous rise in vulnerabilities discovered by X-Force Red annually? One major factor is connectivity.

Connectivity Equals a Wider Attack Surface

The convergence of ICS and the industrial internet of things (IIoT) is one reason why we are seeing ICS vulnerabilities increase. The more pervasive the convergence becomes, the larger the attack surface gets over time.

One example is industrial wireless technologies, or the communication of operational technology with external entities. Think about a simple remote update or support interface, or maybe a private or public cloud, communicating with machinery that was previously disconnected from this type of communication. These technologies and their interaction with IP-based communication channels increase the attack surface and create new attack vectors for industrial controls and the systems they govern.

A common attack vector introduced by many IIoT solutions is the wireless channel used for sensors/actuator communication over a mesh network. A mesh network (aka a meshnet) is a topology used in local networks where the infrastructure nodes connect directly, dynamically and nonhierarchically to as many other nodes as possible. Meshnets are often used when the organization operates IoT and IIoT devices. Unfortunately, this sort of topology can mean that if an attacker gets to one node, they can get to many other nodes on the meshnet and likely automate an attack to sweep through the operational network. Of course, the introduction of an additional attack vector means only that an attacker has a new way to compromise ICS. It does not automatically mean disaster; the system needs to be vulnerable to begin with.

Another reality that can affect these systems’ security is that ICS technologies, such as supervisory control and data acquisition (SCADA), distributed control systems (DCSs), programmable logic controllers (PLCs) and other systems, are typically built, integrated and managed by different entities.

A mixture of different technologies creates a complex infrastructure where multiple vendor products are integrated to keep the plant up and running with minimal interruptions. This inherent complexity and the fact that cybersecurity is, in the majority of cases, not built into products by design or verified during the commissioning phase leaves room for implementation and configuration errors, elevating vulnerability and the risk of an attack.

ICS Security Concerns

ICS clients often ask our team questions such as, “Can someone take over and stop my production processes?” and, “Can a ransomware attack hold my systems hostage?”

Our answers would focus the conversation on the overall risk, and we typically recommend that organizations use a structured and engineered testing approach that includes ICS-specific penetration testing and other techniques to test hardware, software and legacy systems.

ICS penetration testing involves hackers using the same tools, techniques and practices that an adversary would to uncover critical vulnerabilities within ICS environments. The process is preplanned and takes place in a predefined scope and controlled manner. The objective is to find and fix vulnerabilities before attackers can find and exploit them to hurt the organization.

During my many years of testing in various industries and critical infrastructure organizations, I have noticed that security professionals are typically concerned about two things:

1. That the assessment or test itself could impact production systems or safety network components.
2. Most vulnerabilities cannot be easily remediated by the vendors that built the systems and, again, the team aims to avoid impact to production systems because fixing the vulnerabilities may take a long time.

These concerns are legitimate because many critical systems are not designed to be resilient to unexpected and untested anomalies. A security test can potentially generate an anomalous condition, which could affect a system’s stability.

Does this mean that sensitive and critical systems should be left alone and exempt of testing that could destabilize operations? The results of ignoring the potential security issues lurking in the operational environment can end up costing the organization an attack it could have prevented, not to mention the added costs in incident response and disaster recovery from a scenario they never expected.

Possible Solutions: Define Scope and Attack Scenarios and Test

So how can industries relying on ICS tackle this problem? On the one hand, there is the risk of tests bringing down targeted systems. On the other hand, if organizations do not perform testing, sooner or later attackers will discover their soft underbelly and use it to their advantage.

The approach X-Force Red recommends is to structure the test campaign by starting with a defined scope, and then running down a list of potential threats or attack scenarios that are most relevant to the specific environment. The attack scenarios are a list of high-level descriptions of what impact cyberattacks can cause. Aspects such as the architecture, business model, operations, attack surface, level of exposure and technologies in use are all details that can be used to identify applicable attack scenarios, and having a good ICS assets inventory can help a lot here.

Once the list of threats has been defined, a tailored and focused security test can be designed and executed. It’s important to keep in mind that the objective is not to test everything, but instead to focus tests on the subset of critical systems, which are the targets mapped out in the attack scenarios.

Designing and performing the test campaign for these environments is a complex process that must consider aspects such as systems’ intrinsic “fragility” and criticality. It requires extensive experience and deep knowledge of the ICS technology targeted, but it is absolutely possible.

The following testing methods can be executed to verify ICS security levels:

1. Systems and devices configuration checks.
2. Network traffic analyses.
3. Offline vulnerability research.
4. Penetration tests.

Based on the target constraints, organizations should assemble a list of relevant test cases to minimize the risk of instability to tested systems. With this approach, it is easier to keep potential impacts associated with the test’s execution under control.

To summarize, these are the steps we recommend following to help security teams craft their plan before a test:

1. Identify the relevant attack scenarios.
2. Verify your asset inventory.
3. Identify the best testing strategy.
4. Design and evaluate the test cases.
5. Execute the test cases.

Once the security test phase has finished, the vulnerability remediation actions, due to the multiple constraints that these environments have, are often challenging and must be analyzed with great care. For example, installing a security patch or changing a system’s settings should not impact system stability.

The specified attack scenarios can help here too. Organizations should focus on fixing vulnerabilities where possible and on stopping attack chains where that is not possible. Again, experience and a broad knowledge of ICS security solutions, architectures and standards are key to identifying the correct and applicable remediation actions.

Establishing a Testing Program for ICS Environments

X-Force Red recommends establishing a cybersecurity testing program with an ICS security-specialized focus by testers that have the skills and experience to test ICS platforms with an established methodology. The testing program should be designed to cover the most critical ICS components as they apply to the organization’s environment and risk profile.

X-Force Red is comprised of ICS security specialists and testing engineers who are also hackers. They understand how ICS environments operate — including constraints such as the systems that cannot be taken down and that installing a security patch may not be the best solution for remediating a vulnerability — and can help organizations test their environments and remediate according to their specific risk appetite.

Learn more about X-Force Red’s ICS testing services