Imagine running a 911 call center where the switchboard is constantly lit up with incoming calls. The initial question, “What’s your emergency, please?” aims to funnel the event to the right responder for triage and assessment. Over the course of your shift, requests could range from soft-spoken “I’m having a heart attack” pleas to “Where’s my pizza?” freak-outs eating up important resources. Now add into the mix a volume of calls that burnout kicks in and important threats are missed.

That situation is what the average cybersecurity and response analyst deals with.

The concept of “alert fatigue” is real. A March 2023 study commissioned by IBM and completed by Morning Consult found security operation center (SOC) team members are “only getting half of the alerts that they’re supposed to review within a typical workday.” That’s a 50% blind spot.

Moreover, the same study found that most security analysts spend about a third of their typical workday investigating incidents that are not real and that the majority of threats are either low-priority or false positives.

Now, given the above, add in some desensitization to alerts, along with the disjointed, missing, or untested review, response and escalation processes, and the people, process and technology trifecta breaks down. The result? A successful attack.

The cost of delayed detection

A late spring 2023 supply chain attack on a communications provider shows how alert fatigue can contribute to the success of a cyberattack. Go back to the 911 call center: the stream of calls is not stopping, including the “Where’s my pizza?” hysteria. After a while, the call operator zones out. But what if one of those food delivery calls is really something else?

Let us go back to the cybersecurity world for a moment. Just to complicate matters, imagine more “call centers” (or detection tools, such as SIEMs and log aggregators and analyzers) are now triaging the same call but coming to different findings.

Some usual by-products caused by a thwack, or overuse, of similar tools and detection solutions inside the security stack are conflicting results, complacency and information paralysis. All of these issues result not only in delayed detection but also delayed action.

If you are a security operator, have you ever been:

  • Overwhelmed by the level of data to analyze?
  • In a situation that seems to have over-complicated itself?
  • Fearful of making the wrong analysis or taking the wrong action?
  • Faced with too many options and unable to choose?
  • Stuck in the “I need to do more research” feedback loop?

If you have ever experienced these situations, it is quite likely information paralysis has fully kicked in, and response is likely to suffer. So what are the mechanisms to avoid this? Look into the people, process and technology trifecta, and some of the risk management answers lay there.

The right people and processes reduce fatigue

The cybersecurity skill gap, both in terms of available persons and experience, still exists. Therefore, finding the right people is a matter of hiring processes aligning with business needs and ensuring the necessary resources are available. An organization cannot use the “woe is me” line if investments in personnel are not there. Somebody has to run the machines but also give them a reality check.

Where it begins to get challenging are the processes. The processes are very much the bridge between people and technology, enabling sound decision-making. For example, reviewing an alert in isolation from other relevant factors could lead to the wrong action. Context matters. An analyst should not be pulling the fire alarm over a false positive.

Additionally, consider the “who” when it comes to analysis. Is there an escalation matrix available? A peer review process? Anything that does not put the weight of the world on a single person’s shoulders?

As threat investigators try to put the puzzle together, ask these questions. Are we:

  • Getting the right alerts, and in a timely manner?
  • Able to discern where discrepancies are coming from?
  • Missing anything?
  • Able to make a decision based on the information we have?
  • Being kept honest by somebody else?
  • Perhaps most importantly, looking at noise or signal?

Given the overwhelming amount of data and traffic, this is where the right tool stack for you can make all the difference.

The right technology streamlines and triages alerting into digestible pieces

Perhaps the best argument today for integrating more (appropriately tuned!) automation is burnout and fatigue. But remember: automation, orchestration and artificial intelligence, for their own sakes, are not the best investment. That is why “appropriately tuned” is an important caveat.

The most ideal situation is when your SIEM, SOAR and EDR solutions manage the low-hanging fruit. Put another way, the solutions manage the “noise” eating away at your staff’s time so that staff can focus more on the signal. Effectively, your technical solutions are doing the grunt work for you, triaging and funneling what looks to be serious to a human set of eyes. Those eyes — which should have greater context and understanding of the situation at hand — can now focus on the serious work but still maintain the ability to “look back” at what actions the automated tools performed.

To round out the technology piece of this conversation, keep these two thoughts in mind:

  • Less can be more. More tools can create more alerts, including conflicting information. Streamline the technology stack where possible. Some products work well together; some do not. Do your homework for what is right for you.
  • It’s a tool, not a crutch. If your technology solutions lack human oversight, expect a culture of complacency and knowledge reduction to creep in. Do not become dependent on automation, especially if it is only doing what it’s told.

In closing, look to technology not as your savior to the alert fatigue problem. Rather, treat technology as your partner that can do a good deal of the mundane heavy lifting. Do that, and you can focus your time on the threats that matter and ensure you do not fall behind answering those real emergency 911 calls.

More from Incident Response

SIEM and SOAR in 2023: Key trends and new changes

4 min read - Security information and event management (SIEM) systems remain a key component of security operations centers (SOCs). Security orchestration, automation, and response (SOAR) frameworks, meanwhile, have emerged to fill the gap in these capabilities left by many SIEM systems. But as many companies have begun reaching the limits of SIEM and SOAR systems over the last few years, they have started turning to other solutions such as extended detection and response (XDR). But does this shift spell the end of SIEM…

X-Force releases detection & response framework for managed file transfer software

5 min read - How AI can help defenders scale detection guidance for enterprise software tools If we look back at mass exploitation events that shook the security industry like Log4j, Atlassian, and Microsoft Exchange when these solutions were actively being exploited by attackers, the exploits may have been associated with a different CVE, but the detection and response guidance being released by the various security vendors had many similarities (e.g., Log4shell vs. Log4j2 vs. MOVEit vs. Spring4Shell vs. Microsoft Exchange vs. ProxyShell vs.…

Unmasking hypnotized AI: The hidden risks of large language models

11 min read - The emergence of Large Language Models (LLMs) is redefining how cybersecurity teams and cybercriminals operate. As security teams leverage the capabilities of generative AI to bring more simplicity and speed into their operations, it's important we recognize that cybercriminals are seeking the same benefits. LLMs are a new type of attack surface poised to make certain types of attacks easier, more cost-effective, and even more persistent. In a bid to explore security risks posed by these innovations, we attempted to…

Bringing threat intelligence and adversary insights to the forefront: X-Force Research Hub

3 min read - Today defenders are dealing with both a threat landscape that’s constantly changing and attacks that have stood the test of time. Innovation and best practices co-exist in the criminal world, and one mustn’t distract us from the other. IBM X-Force is continuously observing new attack vectors and novel malware in the wild, as adversaries seek to evade detection innovations. But we also know that tried and true tactics — from phishing and exploiting known vulnerabilities to using compromised credentials and…