Threat Hunting Services Are Now a Basic Necessity

For the past few years, the security industry has seen a gradual move away from traditional, resource-heavy endpoint protection agents to next-generation solutions in response to the increasing sophistication of malware, cybercriminal tactics and the threat landscape at large.

Traditional host intrusion detection systems (HIDS) built on signature-based detection rules are no longer sufficient to combat techniques such as living off the land, where an attacker uses readily available tools on corporate systems to move laterally within the network or exfiltrate information, and fileless malware, where malicious binaries aren’t written to disk.

Threat Hunting With Endpoint Detection and Response

Endpoint detection and response (EDR), first coined by Gartner back in 2013, provides a solution to this problem by continuously recording all endpoint activity using lightweight agents running on the endpoints and passing it to a centralized location. This pool of data allows analysts to automatically search for activity involving known indicators of compromise (IoCs) such as binaries, domains and IP addresses.

It also enables them to track down suspicious activity from known good applications, search through unaltered historical data after an attacker erases log information on the endpoint itself and find every change a binary makes to the system. By integrating this information source into your existing security information and event management (SIEM) platform and correlating with other log sources, analysts can fill in an important security information gap.

EDR solutions are immensely valuable tools for security analysts and threat hunters because they facilitate tracking and deep analyses on anything that falls outside of normal day-to-day activity. As a result, it becomes much harder for attackers or malware to stay under the radar.

Listen to the podcast: The Art of Cyber Threat Hunting

EDR in Action

An interesting example of how EDR tools bridge information gaps became apparent during the recent Petya/NotPetya malware outbreak. Traditional signature-based endpoint detection triggered alerts based on the malicious nature of the malware’s binaries and revealed only the host that was potentially infected. However, EDR tools allowed threat hunters to determine what files the malware wrote to disk or attempted to modify. These tools also revealed how the malware spread laterally across the network, scanned the subnet for additional targets, renamed versions of common tools such as psexec.exe and deleted local event logs in an attempt to erase its tracks. Most importantly, EDR tools enable analysts to look forward by building new detection rules based on findings and fine-tuning these rules to act on very specific behavior.

In the case of Petya/NotPetya, knowing exactly which files were written to disk and what common tools were being leveraged, as well as understanding the full process flow, allowed threat hunters to use specific parts of the analyzed information to build new behavioral detection rules to all hosts running the EDR agent. This ensured that any endpoint showing similar activity was immediately identified, allowing customers to hit the ground running with all the details necessary to remediate.

A Deep-Dive Hunt for Petya

A deep-dive threat hunting session for the Petya campaign using Carbon Black Response as the EDR solution uncovered the following information in our lab environment. It also provided us with a clear process tree showing exactly how the malware executed and spread across the endpoints.

  • The Carbon Black Reputation threat intelligence feed and any feeds based on a VirusTotal score generated alerts for a particular, randomly named file (xxxx.tmp, MD5 7e37ab34ecdcc3e77e24522ddfd4852d), which was dropped and executed as part of the malware execution flow. This provided a starting point for the analysis.
  • The unsigned binary xxxx.tmp opened a handle with change rights into lsass.exe — behavior often seen with tools such as Mimikatz to steal user credentials.
  • A file named dllhost.dat (MD5 aeee996fd3484f28e5cd85fe26b6bdcd) — which, based on the signature information, turned out to be a renamed version of the Sysinternals psexec tool — was dropped. This binary was used to spread the malware to other hosts on the network, as indicated by the command line, ‘C:\Windows\dllhost.dat \\10.239.15.217 -accepteula -s -d C:\Windows\System32\rundll32.exe “C:\Windows\027cc450ef5f8c5f653329641ec1fed91f694e0d229928963b30f6b0d7d3a745.dll”,#1 54’.
  • Event logs and the update sequence number (USN) change journal were deleted (‘/c wevtutil cl Setup & wevtutil cl System & wevtutil cl Security & wevtutil cl Application & fsutil usn deletejournal /D C:’).
  • A scheduled task was configured to reboot the host (‘/c schtasks /Create /SC once /TN “” /TR “C:\Windows\system32\shutdown.exe /r /f” /ST 08:57’).
  • The single rundll32.exe parent process for all the activity described above made a large amount of outbound network connections to ports 445 and 139, indicating an attempt to spread to other hosts. It also made several modifications to a wide range of file types (e.g., PDF. txt, zip, xls, doc, 7z).

Creating Automatic Alerts

Based on these events, we can use Carbon Black Response to isolate the infected hosts from the network and allow forensic analysis. We can also ban malicious hashes (e.g., MD5 7e37ab34ecdcc3e77e24522ddfd4852d) and prevent those binaries from executing on any other endpoints running the EDR agent. Additionally, we can create new search queries that will automatically alert us to typical behavior discovered during the analysis. Going forward, we might want to alert on processes or binaries that:

  • Match a signed process named dllhost.dat with a digital signature by “Microsoft Corporation” (process_name:dllhost.dat digsig_publisher:”Microsoft Corporation” digsig_result:signed);
  • Match binaries with psexec Sysinternals signature data but are not named psexec.exe or psexesvc.exe (product_name:’Sysinternals PsExec’ company_name:Sysinternals -observed_filename:psexec.exe -observed_filename:psexesvc.exe);
  • Attempt to use wevtutil to clear event log entries and fsutil to delete the USN change journal (parent_name:rundll32.exe process_name:cmd.exe cmdline:fsutil cmdline:deletejournal cmdline:wevtutil);
  • Add a scheduled task to reboot the host (parent_name:rundll32.exe process_name:cmd.exe cmdline:shutdown.exe cmdline:schtasks);
  • Are named rundll32.exe and make over 40 network connections, connect to either port 445 or 139, and make changes to more than 50 files (process_name:rundll32.exe AND netconn_count:[40 TO *] AND filemod_count:[50 TO *] AND ipport:445 OR ipport:139);
  • Have a filename ending in .tmp, which open a handle into lsass.exe (process_name:*.tmp crossproc_name:lsass.exe); or
  • Contain activity involving the MD5 hash 7e37ab34ecdcc3e77e24522ddfd4852d.

Notable MD5 hashes observed during the analysis include:

  • xxxx.tmp: 7e37ab34ecdcc3e77e24522ddfd4852d; and
  • dllhost.dat (psexec.exe): aeee996fd3484f28e5cd85fe26b6bdcd.

Get the Most Out of Your EDR Solution

As with any tool, the value EDR adds is based on how much of its functionality is actually leveraged. Often, IT teams lack the resources to continuously hunt threats or conduct eyes-on-glass monitoring.

A managed security service provider (MSSP) can handle the required 24/7 monitoring and managed SIEM services. It can also provide the necessary threat hunting skills to help you fully leverage the functionality of EDR solutions. This allows you to be notified of any malicious or suspicious activity on your network as soon as it occurs, and the additional threat hunting can provide you with the deep-dive analysis required to pinpoint security weaknesses and generate specific recommendations on how to fix these shortcomings.

Learn More About Managed Detection and Response Services

Share this Article:
Thomas Bouve

Senior Threat Response Analyst, IBM

Thomas Bouve is a Senior Threat Response Analyst with IBM's Managed Security Services. Prior to joining IBM, he worked as a system administrator for a leading Belgian university and a number of tech startups. He started his career with IBM as an analyst for the Threat Monitoring team and currently spends his days as a threat hunter for X-Force Services, researching malware and analyzing advanced threats.