When a breach is uncovered, the operational cadence includes threat detection, quarantine and termination. While all stages can occur within the first hour of discovery, in some cases, that’s already too late.

Security operations center (SOC) teams monitor and hunt new threats continuously. To ward off the most advanced threats, security teams proactively hunt for ones that evade the dashboards of their security solutions.

However, advanced threat actors have learned to blend in with their target’s environment, remaining unnoticed for prolonged periods.

Based on years of experience, here are some of the threat-hunting basics used by the IBM X-Force team to find these actors.

What threat hunting is

Threat hunting is a proactive approach to identifying previously unknown or ongoing non-remediated threats within an organization’s network. Threat hunting should be iterative and human-driven.

Effective threat hunting requires a specific skill set. A successful threat hunter must be good at hypothetical thinking and be able to speculate about source vectors and potential impact.

Additionally, pattern recognition and deductive reasoning are valuable skills for the job. Attackers are constantly getting better at finding new, creative ways of exploiting weaknesses in operating systems and applications. That’s why threat hunters must look for patterns matching tactics and unusual behavior.

It’s essential to formulate and develop logical theories on how to access a network or exploit a system to gain access to critical information. Once the theory has been created, an analyst needs to work backward, using deductive reasoning, to look for any clues left behind by attackers.

Additionally, threat hunting is an iterative process. A good threat hunter must be able to quickly repeat the same steps because similar attacks are likely to happen again.

But most importantly, threat hunting is proactive. Overreliance on alerts alone can ultimately lead to tunnel vision.

What threat hunting is not

Just as it is essential to define what threat hunting is, it’s helpful to understand what it is not, according to Neil Wyler, global lead of active threat assessments at IBM X-Force. Threat hunting is not:

• Ctrl+F Indicator of Compromise (IOC) — Threat hunting isn’t hitting a “Ctrl+F” IOC to locate threats. You should look at the tactics, techniques and procedures that attackers use in your environment. When the hunt is in progress, you shouldn’t be looking for an IP address, a hostname or a file hash — these elements are lower on the pain pyramid for threat hunting. You need to find the evidence that tactics and tools leave behind.
• Automated — Automation can certainly help once threats or datasets of interest are identified, but it isn’t a starting point.
• New — Threat hunting, in some form, has existed for many years, whether looking for outliers in data logs or sorting by count in Excel spreadsheets.
• Magic — Threat hunting is something even entry-level analysts can do. You may only know some things about it. But if you have an inquiring mind and ask the right questions, your expertise will develop quickly.

Why hunt threats?

Threat hunting is important. Effective threat hunting helps reduce the time from intrusion to discovery, minimizing the damage done by attackers. The longer the time lapse between system failure and response, the more damage the organization suffers during an attack.

Additionally, threat hunting can help you:

• Find previously undetected threats and reduce dwell time (infection to detection).
• Understand your security environment to enhance the speed and accuracy of response. This will provide you with a considerable advantage over an attacker.
• Improve overall organizational posture. Don’t wait for an alert to go through your security information and event management (SIEM) tool. Find misconfigurations, identify gaps and help reduce attack surfaces quickly.

Nothing is more valuable than learning from real-life situations and years of hands-on experience. If you’ve been in security for a couple of years, you know how difficult it is to get security analysts, architects and threat hunters to share their knowledge. Black hat hackers use underground forums to exchange their insights all the time. That’s how they evolve and improve their tactics. So why can’t we, a force for good, get into that habit?

Here are some threat-hunting best practices demonstrated in a real-life story.

To Russia, with love

The client who hired a threat-hunting team was one of the largest financial organizations in the world, with a mature security practice and employing hundreds of security professionals. As the hunt formally began, the director of security ran the visiting threat hunters through their security framework.

Threat hunting can be compared to playing the game of outliers. When you look for network traffic, you should pay attention to the lowest number of sessions for a particular protocol. You should ask yourself: Why do these sessions exist?

The threat hunters decided to look at the file transfer protocol (FTP). According to the director of security, the organization never used FTP. But there was a small amount of FTP traffic in this environment. Threat hunters then went through it manually and looked at the clear text protocol.

It turned out the FTP connection was going to the .ru host in Russia. The company was unfamiliar with that traffic, prompting the team to look deeper into the matter. What they discovered was quite shocking. At 1 a.m. every night, files were transferred to an FTP server over clear text protocol to a .ru host.

Threat hunters then did a quick file extraction and discovered that the leaked files contained every financial transaction and trade the company had made in the past 24 hours. These files had been sent out daily for the last six months — unnoticed by the victim.

The company in this story had a robust security setup — packets, logs and endpoint behavior analytics. Their team consisted of hundreds of seasoned security professionals. But everything was reactionary. Their team was waiting for an alert to begin their work.

The behavior analytics software that the company used was installed when the attacker was already in the environment. As a result, the attacker’s communication over a protocol was baselined as usual.

The moral of this story is that human-driven security matters. If your team deploys software, humans need to question and verify processes. That company learned a lesson about the danger of tunnel vision: they never investigated something suspicious that they thought they’d blocked. And even with the most comprehensive technology in place, your team needs to be proactive to ensure your security program is mature.

Threat hunting problem areas

Most real-life threat-hunting stories have commonalities and patterns that could be summarized in the following problem areas:

• Tunnel vision — Leave bias at the door and be objective with your investigation. Tunnel vision can cause you to ignore an attacker’s presence.
• Internal threats — Remember to monitor behavior inside your network even though your priority is external threats. Complacency about internal threats can lead to a single forgotten web server being weaponized by the adversary.
• Poor access control — Segmentation and encryption play a significant role in threat hunting. Don’t make it easy on attackers.
• Direct-to-IP communication — Ensure you’re configuring your virtual private network clients correctly. Know what’s on the other end of that IP address. Build feeds and tie them to resources.
• No host isolation — Trust nothing. You’re not always on the corporate network. Segment your networks and isolate your hosts.

The 5 golden rules of threat hunting

Do you know what to look for and where to look? Here are five rules for threat-hunting success:

1. Collect logs from key areas.

Logs are critical to threat hunting. Collect logs from your key areas, including switches, routers, firewalls, proxies, web servers, applications, operating system events, PowerShell commands, audits and EMETs. You don’t have to send them to your SIEM but at least consider writing them to disk.

2. Monitor network data.

Know your environment’s data ingress and egress points. Know how your subnet roles are set up and establish directionality. Do full packet capture with a minimum of three days raw and two months of metadata.

3. Analyze endpoint behavioral data.

Inventory all processes, scheduled tasks, unexpected services, registry access, and file and network data. Make sure you are hunting Amcache and Shimcache — gold mines of interesting data. Understanding what looks normal will help you identify when an anomaly occurs.

4. Practice situational awareness.

After you aggregate all that data, what do you do? You use situational awareness.

• Understand what normal looks like on your hosts and network. Create a baseline for comparison.
• Become aware of what is normal so that when an anomaly occurs, it sticks out like a sore thumb.
• Outliers are always interesting though not always evil. Every time you see something, it doesn’t mean it’s an attacker. But you’ll learn something from that experience.

5. Leave preconceived notions at the door.

Threat hunting isn’t magic. Spend time getting better at what you do. Don’t start with an IOC. Start with a question: if data were leaving the environment, where’s the most likely place it would go? How would I get in from the outside if I were an attacker?

Learn more

Threat hunting is a straightforward way to detect malicious activity before damage occurs. The bottom line is that even if a data breach hasn’t affected you yet, it will the future. That is why all organizations should be hunting for threats using human analysts.

Security is a team effort. With the help of purpose-built technology and integration, you can fully control your environment. Learn about IBM Security’s threat intelligence integrations on IBM Security App Exchange. And if you’d like to share your thoughts about threat hunting, be sure to share them with the security community.

Want to hear directly from the experts? Schedule a no-cost consult meeting here: IBM X-Force Scheduler.