How much information do you need to manage your security? Until recently, the answer was always “more.” That is changing as the sheer volume of available data grows.

Today, corporate networks and machines are significantly faster and more powerful than they were 10 years ago. Combine this advancement with threat intelligence feeds, which include logs from new applications and streams from hosted services, and you have a surge in security data. This dramatic increase of data has gradually become a hindrance to organizations.

There is also a conflict between two different objectives for the information being gathered. First, timely incident response and reduced dwell time call for the acquisition of a limited amount of data to identify and stop attacks as quickly as possible. Second, system recovery and forensic examinations consider as much data as possible once the attack is over. The security industry has taken a one-size-fits-all approach, miring teams in unnecessary data and forcing them to look for real-time indicators when they should be looking for root causes.

Don’t Drown in Security Data

A recent survey by Enterprise Security Group (ESG) highlighted the increased volume of this information as a problem for the majority of participants. The skyrocketing number of devices and amount of network traffic is actually making the job of securing the organization more difficult. In fact, this growth poses an even greater risk than the increasing sophistication of the attacks themselves.

Cognitive security improves the ability of analysts to make sense of data in context, but the sheer volume of events can be overwhelming. It is time to re-evaluate the information gathered and stored by this process. To counter the negative effects of the irreversible growth in the volume and complexity of these inputs, we must approach security data differently.

Transient Versus Long-Lived Data

Not all security monitoring events are created equal when viewed through the lens of their usefulness and half-life. Take an infected endpoint, for example. For the purpose of rapidly identifying and interdicting an attack, a limited set of indicators of compromise (IoC) is likely sufficient to recognize that a machine has been corrupted and raise the alarm. For the purpose of reconstructing the event and hopefully restoring the victim’s machine, all the discrete actions within that attack need to be recorded and played back.

In the first case, a minimal set of information is gathered to identify an attack in progress, since the priority is arresting the attack in flight. Too much data would slow the process and increase the likelihood of spreading the infection. The second case, which prioritizes recovery and shortened downtime, requires a much more detailed set. The simpler set of IoCs would be insufficient to reconstruct the event or the system.

Both sets of data are optimized for their respective purposes and are not interchangeable. As a result, there must be different policies and even platforms for gathering the requisite information to support business needs.

Is the Data Useful?

As the sources of security monitoring information grow, it’s important to conduct periodic reassessments and pruning to eliminate what is redundant and prioritize what is required. If network-based detection of data exfiltration is approaching zero due to the installation of strong host-based data loss prevention (DLP), for example, network-based information gathering can be relaxed. If host-based intrusion detection is no longer catching infections because of stronger gateway filtering and endpoint management, analysts should deprioritize intrusion detection system (IDS) data accordingly.

The pruning also addresses a real risk in the creation and storage of all this data. Once the systems are created, the organization is responsible for reviewing and addressing data. During one major breach in 2014, a strong security team with a sizable investment and considerable experience received public criticism for not responding to security messages that were generated in the early stages of the attack.

Gathering as much data as possible creates a mass of information. This may be useful for root cause or forensic examination, but it also decreases the likelihood of observing an event as it occurs.

Assessing Business Need

Depending on the role an organization plays in the business, the relative priority of detection and recovery differs. When this happens, information gathering characteristics should change as well. An internal-facing organization with no customer information or external contact will not have the same public compliance and notification concerns as a customer support organization that handles partner records or personally identifiable information (PII). When an organization is held to a regulatory or contracted standard, the relative values of detection and recovery are not the same.

Organizations will also be concerned with different types of threats. Some will be more worried about long-running advanced persistent threats (APTs), data leakage and credential theft, while others are vulnerable to urgent, immediate attacks such as ransomware and distributed denial-of-service (DDoS). For the former, detailed recovery information or detection over a long period and across multiple machines is more valued than the immediate identification of a seriously impacted network or the isolation of a disabled and compromised host. The latter requires rapid identification and even real-time data.

Step Up to the Challenge

This appetite for security event data has been created and whetted by a universal understanding that security will never be 100 percent effective. Unfortunately, this has led to a decreased urgency to inch closer to total security in spite of this growing complexity in detection and response.

Even the mere threat of attack can lead to a singular focus on a strong monitoring and response infrastructure and an overall sense that recovery is possible. It underestimates the irreparable damage some attacks cause and creates an imbalance. Decreasing dwell time is very important, but it should be secondary to decreasing the likelihood that target systems will be breached.

Security analysts must step up to the challenge of integrating the ever-increasing volume of security data into better security management and expanding monitoring. This challenge is not new, but it is rapidly becoming more burdensome. It’s time to think creatively about applying and understanding the security data we encounter every day before the new wave of information swamps us.