It sounds simple: A scanner identifies a vulnerability, the vulnerability is patched. What happens in between, however, can be far from simple. Yet if you are not on a security team or, more specifically, a vulnerability management team, you would never know the bumpy, winding road that often stretches between scanning and patching.

The Patch Management Process

When vulnerabilities are discovered within applications, networks, systems and other parts of an organization’s environment, the priority for those managing them is to ensure they are patched and up to date. It’s an essential practice that helps maintain security over time and reduces the risk of an attack. But while patching vulnerabilities is a basic concept, the process of patching tends to run on a separate track from the information security team’s, and is often only visible to individuals directly involved with it.

The result is an oversimplification of the patching process. Questions from outside the security organization may surface, such as, “Why not just patch it?” Alas, when it comes to patching, the devil is in the details. Business considerations, such as keeping certain systems running and avoiding technical hiccups like applied patches that do not work, can slow down and temporarily halt the process.

To understand the top issues organizations may face between scanning and finding a vulnerability and actually patching it, I spoke with X-Force Red Hacking Chief Technology Officer (CTO) Steve Ocepek. Steve and his team help organizations around the world fix their most critical vulnerabilities.

I asked Steve to explain the top five bumps in the road that pop up during the process of patching. Here is what he shared.

1. The Business Leader Versus Security Leader Conundrum

What if a patch causes a system that serves thousands of clients to go down? That sort of scenario is most companies’ worst nightmare. More often than not, business continuity trumps vulnerability and patch management.

You may have heard of the term CIA. No, it’s not that government organization you are thinking of. The CIA Triad in information security stands for confidentiality, integrity and availability, which describe the goals related to the systems and data that we use. Those three concepts are at the top of both security and business leaders’ minds.

Running a business includes the availability of systems to ensure its continuity. Depending on the types of services they offer, some organizations are more concerned about availability than confidentiality or integrity. Therefore, when a chief information security officer (CISO) asks their team to push a patch, the business leader who owns the system that needs patching may reply, “You cannot touch those boxes. We are committed to a 99.999 percent uptime on those, and we will lose money in the five-figure category for every hour they are down. Fill out these forms first and prove you won’t take the system down.”

This sort of conundrum can slow down the patching process, leaving the system exposed to attackers for months or even years. Many times, they are not patched until it is too late, and beyond.

2. The Patch That Won’t Patch

Even when a decision comes through to patch immediately, not all patches work. Sometimes a remediator will apply a patch, only to find that the vulnerability remains unpatched after the next scan. This typically happens when a patch comes with a file or document providing instructions for applying the patch correctly, and that document is ignored or skimmed at best. For example, the instructions may require someone to manually change a setting before deploying a patch. Remediators may overlook those instructions, which leads to an incomplete patch.

Repeating that process without reviewing the documentation can lead to a vulnerability that keeps surfacing in a scan and remains unpatched.

3. False Positives

Let’s face it: Not all signatures that come out of the box in vulnerability scanners fit all. Bad ones exist, or at least ones that do not suit the organization’s needs, and they can cause vulnerability scanners to misunderstand what they see.

For example, a scanner might scan a web server running on a physical box. The scanner uncovers a vulnerability in the server’s operating system (OS); however, the vulnerability can only be exploited by an attacker if a certain function is enabled. The scanner cannot determine whether the organization’s environment has that function enabled, but it still flags the vulnerability. As a result, the security team may waste time chasing down that vulnerability, when in reality it’s not a real vulnerability because the required function is not enabled.

From a vulnerability management team’s point of view, it can be challenging to prove which vulnerabilities are real vs. false positives. They could easily waste time chasing down false positives for patching instead of focusing on the ones that really matter.

Based on what Steve and his team experience when working with vulnerability management teams, 20-30% of the team’s time is spent on false positives.

4. Third-Party Vendor Vulnerabilities

Some vulnerabilities are the kind that companies cannot fix themselves, such as those exposing technologies hosted by third-party vendors. Firewalls, internet-connected cameras and other technologies may be in that scanning scope and may contain vulnerabilities that will be hard or impossible to patch.

Security leaders can report their findings to the third-party vendor and ask that it patches the issues. However, that does not necessarily mean the vendor will fix them promptly, or at all. So, what can security teams do? One solution in such cases is to apply compensating controls and segment those devices from the rest of the network so that if a vulnerability is exploited, the attacker won’t be able to leverage it to access the company’s infrastructure.

You can also turn off features on vulnerable devices to limit the overall exposure and make them less risky to operate.

5. Asset Inventory Blind Spots

This is one of the most common issues Steve and his team witness in the process of vulnerability scanning. A scanner finds a vulnerability on a given system, yet the security team had no idea that the system in question even existed on the company’s network, let alone who owns that system. The issue is a result of asset inventory being incomplete or out of date, which is, unfortunately, a very common scenario.

Steve and his team have yet to meet a client who has said, “Our asset inventory is perfect!” The reason why this problem is so prevalent is because the asset database concept is outdated. In today’s dynamic environment, with millions of devices and endpoints connecting to the network, maintaining an asset inventory is nearly impossible. Therefore, uncovering a vulnerability on an unrecognized system is commonplace and can be a headache for the security team.

It means discovering the asset, obtaining information on its purpose, its owner, the data and functions it hosts, the controls that may be applied to it, and, eventually, the vulnerabilities that have been exposing it to a potential attack. These blind spots can definitely add delays to the patching process of out-of-scope assets.

Good Patch Management for Better Security

In an era of attacks that can wreak havoc across the entire globe, patching is a must-do despite these bumps in the road. The good news is that most of these issues are easily solvable. In some cases, it can be an easy fix, at no extra cost, such as segmenting third-party devices or making sure your remediators read and apply all patching instructions. To avoid running into business continuity issues, you should also abide by the time slots allocated for patch management to minimize the risk of business-critical systems going down and negatively impacting the organization.

To better manage patching priorities, you may want to consider using an automated ranking formula that prioritizes vulnerabilities based on those that are actively being exploited by attackers.

Partnering with a team of hackers such as X-Force Red can also help. Our team runs a vulnerability management program that includes facilitating the remediation process. That means that if there are bumps in the road, we can help overcome them.

For example, if a vulnerability surfaces on a system that is unknown to the client, we can help track down that system and the department that owns it, and then push the vulnerability to the system owner and confirm it is remediated. To work through business continuity issues, our team works with security and business leaders to understand how and when to apply patches so that systems in production are not disrupted. Finally, we can also manage the scanning process, which can include running the scan results through our automated ranking formula and weeding out false positives before they are sent to your vulnerability management teams.

Considering we think like hackers and find vulnerable systems on a regular basis, we can help detect problems on assets whether they are known or unknown to the client.

To learn more about the top obstacles that may emerge during the patching journey and how to overcome them, join our X-Force Red webinar on Oct. 15 at 11 a.m. ET. Register today.

Learn more about X-Force Red Vulnerability Management Services