Your security incident detection capabilities are at the heart of your organization’s incident response plan. After all, if you are unable to recognize incidents, it is not possible to start an incident response plan. Although incidents can occur in countless ways — and it’s impossible to detect every possible attack scenario — you need to develop a strategy for testing and improving your existing detection capabilities that incorporates methods ranging from testing on paper to running a full-blown cyber simulation.

Your testing strategy should go hand-in-hand with general efforts to improve your team, and it should also mandate sufficient quality checks. Just as you wouldn’t want your antivirus to trigger on benign, nonmalicious files, you wouldn’t want to start an incident response plan based on insufficient or faulty detection capabilities.

Traditional Testing Strategies

Paper Tests

A paper test is what it sounds like: a test on paper. It’s often the first step in your strategy, and its output is used to develop incident response plans. You can start by identifying your key business assets and data flows and documenting your existing detection capabilities. Some key points to consider include:

  • How easy would it be for malicious actors to bypass your detection capabilities?
  • How reliable, complete and accurate is your information?
  • Do you have direct access to the detection information, or is the information curated first? Assessing any assets that are not under your control can be a challenge. There’s a big difference between having direct access to log events and having to rely on a weekly report from a service provider.

For a paper test, you can start by simulating data flows and describing where and how you intend to detect malicious activity. There are limits to this method, though, as this kind of test leaves considerable room for error and typically should only be used in the initial overview of your capabilities.

Tabletop Exercises

A more structured approach utilizes a tabletop exercise that would allow your team to produce a cyber simulation on paper and then review and practice your response to the situation at hand. While a paper test is often more focused on one asset, a tabletop exercise takes a more holistic approach that can show you the areas of business (or logic) where your detection coverage could be improved.

A tabletop exercise can be conducted with just technical personnel, but the simulation tends to be most effective if you have stakeholders from different departments participating.

Cyber Simulation Datasets

Different Types of Datasets

The next step is to use datasets for cyber simulation. A common approach involves developing simulations based on the content of linked datasets, inputting the datasets into your security solutions and checking if they trigger alarms. For example:

  • Use datasets with network information to test the detection rules of your intrusion detection system (IDS):
    • One of the largest collections is at, which has captures of malspam, malware and ransomware infections. Other sources include the Shadowbrokers PCAPs with traces of EternalRomance and DoublePulsar and a collection of malware traffic by Contagio.
    • PCAP files can be replayed with tcpreplay or ingested directly by your IDS.
  • Scan host-based artifacts to test endpoint protection:
    • Start by downloading the EICAR or WICAR test samples and using repositories from Malshare, VirusShare or theZoo.
    • You do not have to execute the samples — the presence of the file should be enough to trigger detection. Take care though, and do this only in a controlled environment.
  • Load system and application logs to tune a security information and event management (SIEM) solution:
    • The datasets of Los Alamos National Laboratory‘s network can enable you to verify that your SIEM can distinguish between malicious and nonmalicious traffic. The set of Windows event log files associated with specific attack and post-exploitation techniques can help you verify that your SIEM is able to spot malicious behavior.

Harness Your Datasets

There are a number of key points to consider as you use these simulation datasets. First, you should verify that your security device actually alerts on elements’ datasets. Take into consideration whitelists that are applied and the limitations of detection. If a detection is based on elements in a log file, then verify that those elements (or events) are at least logged. A good example here is Sigma rules, which rely heavily on logged Windows events.

Next, check that the alert is complete and provides sufficient information for further investigation. The alert should at least contain a timestamp, the reason for alerting and the affected asset. Then you can verify that the detected alert is accurate. This means that detection should only trigger on malicious activity and shouldn’t cause too many false positives. To tune your rule set, you will need a corpus of “goodware.” For example, for YARA, you might use the retrohunts on goodware feature by VirusTotal.

Finally, make sure that the alert and the associated artifacts can be processed by your forensic investigation tools so you can follow up with your incident response. One way of preparing these tools is to test them against reference datasets —like the Computer Forensic Reference Data Sets by NIST or the disk images by Digital Corpora — before the fact so you understand their limitations and strengths.

Explore Advanced Simulation Datasets

If you have explored these datasets, you can move forward and use more advanced datasets, such as the set from Splunk, Boss of the SOC (BOTS), which combines Windows events, IDS events and application data. Other datasets include repositories of public exploits from malSploitBase or exploit-db (which can help you test both network and endpoint detection capabilities) and the Atomic Red Team, which can help test endpoints.

One dataset worth mentioning is traffic captures that are made available by malware analysis sandboxes, either internal or public. For example, the malware sandbox Hybrid Analysis provides not only PCAP files but also samples of other related malware files. You do need to sign up first. Descriptions of detection methods listed among the MITRE ATT&CK techniques can be sources of inspiration as well.

Additionally, you can always create your own datasets by using tests included in Metasploit or by crafting your own network traffic via pCraft. But crafting your own network traffic is only the beginning — the next step is emulating adversary behavior. You can do this via blue team/purple team exercises, but this is only really effective if you have well-established incident response processes already. Another potential approach is to develop your own scenarios. As it happens, Caldera by MITRE builds on the ATT&CK framework and is fit for this job.

Automate Your Detection Capabilities

The holy grail of verifying your detection capabilities is automated adversary emulation. The Caldera framework built by MITRE is a perfect fit, but it does require involvement from your team. The biggest advantage is that it’s built on the MITRE ATT&CK framework, so you can reuse it to demonstrate your detection coverage later on.

Improving your detection capabilities can start with a simple paper list that documents strengths and weaknesses. Once you have produced this, you can challenge your readiness via tabletop exercises and certify your protections with bits and bytes via cyber simulation datasets.

More from Intelligence & Analytics

Hive0051’s large scale malicious operations enabled by synchronized multi-channel DNS fluxing

12 min read - For the last year and a half, IBM X-Force has actively monitored the evolution of Hive0051’s malware capabilities. This Russian threat actor has accelerated its development efforts to support expanding operations since the onset of the Ukraine conflict. Recent analysis identified three key changes to capabilities: an improved multi-channel approach to DNS fluxing, obfuscated multi-stage scripts, and the use of fileless PowerShell variants of the Gamma malware. As of October 2023, IBM X-Force has also observed a significant increase in…

Email campaigns leverage updated DBatLoader to deliver RATs, stealers

11 min read - IBM X-Force has identified new capabilities in DBatLoader malware samples delivered in recent email campaigns, signaling a heightened risk of infection from commodity malware families associated with DBatLoader activity. X-Force has observed nearly two dozen email campaigns since late June leveraging the updated DBatLoader loader to deliver payloads such as Remcos, Warzone, Formbook, and AgentTesla. DBatLoader malware has been used since 2020 by cybercriminals to install commodity malware remote access Trojans (RATs) and infostealers, primarily via malicious spam (malspam). DBatLoader…

New Hive0117 phishing campaign imitates conscription summons to deliver DarkWatchman malware

8 min read - IBM X-Force uncovered a new phishing campaign likely conducted by Hive0117 delivering the fileless malware DarkWatchman, directed at individuals associated with major energy, finance, transport, and software security industries based in Russia, Kazakhstan, Latvia, and Estonia. DarkWatchman malware is capable of keylogging, collecting system information, and deploying secondary payloads. Imitating official correspondence from the Russian government in phishing emails aligns with previous Hive0117 campaigns delivering DarkWatchman malware, and shows a possible significant effort to induce a sense of urgency as…

X-Force releases detection & response framework for managed file transfer software

5 min read - How AI can help defenders scale detection guidance for enterprise software tools If we look back at mass exploitation events that shook the security industry like Log4j, Atlassian, and Microsoft Exchange when these solutions were actively being exploited by attackers, the exploits may have been associated with a different CVE, but the detection and response guidance being released by the various security vendors had many similarities (e.g., Log4shell vs. Log4j2 vs. MOVEit vs. Spring4Shell vs. Microsoft Exchange vs. ProxyShell vs.…

Topic updates

Get email updates and stay ahead of the latest threats to the security landscape, thought leadership and research.
Subscribe today