“Experience is a hard teacher because she gives the test first, the lesson afterwards.” – Vern Law

The penetration test is an important tool to help organizations assess the state of their defenses. The National Institute of Standards and Technology (NIST) defined a pen test as “a test methodology in which assessors, using all available documentation (e.g., system design, source code, manuals) and working under specific constraints, attempt to circumvent the security features of an information system.”

Why Are Penetration Tests Necessary?

While it may be tempting for business executives to look at a pen test as a hard measure of the organization’s security defenses, not all pen tests are created equal. Even NIST warned that “penetration testing represents the results of a specific assessor or group of assessors at a specific point in time using agreed-upon rules of engagement.”

Much depends on how the pen test is scoped. NIST cautioned that “considering the complexity of the information technologies commonly employed by organizations today, penetration testing can be viewed not as a means to verify the security or privacy features of an information system, but rather as a means to: (i) enhance the organization’s understanding of the system; (ii) uncover weaknesses or deficiencies in the system; and (iii) indicate the level of effort required on the part of adversaries to breach the system safeguards.”

Compared to a vulnerability scan, a penetration test provides “an explicit and often dramatic proof of mission risks and an indicator of the level of effort an adversary would need to expend in order to cause harm to the organization’s operations and assets, to individuals, to other organizations.”

However, the test also provides something that is often missing when reviewing the effectiveness of security spending: an assessment of how well the organization is able to respond to an attacker — albeit an attacker whose rules of engagement are well-defined.

Read the interactive white paper: Preempt attacks with programmatic and active testing

Scoping Is Key

Determining the correct scope and parameters is key to ensuring the organization gets good value out of this expenditure. An overly broad pen test will cost more and is likely to disrupt business systems, as well as other systems that may not house any sensitive or valuable data. On the opposite end of the spectrum, a narrowly scoped test may only focus on a few key systems and miss other relevant systems that house critical data (e.g., operations data, intellectual property data, HR data, HIPAA data and payroll data).

In its Penetration Testing Guidance report, the PCI Security Standards Council cautioned that the organization should work closely with the pen tester during scoping “to verify that no components are overlooked and to determine whether any additional systems should be included in scope. The scope of the penetration test should be representative of all access points, critical systems and segmentation methodologies” for the system(s) covered.

While 10 years ago it might have been acceptable for a business executive to mistake a successful PCI pen test to mean that all the organization’s systems were safe, doing so in 2016 is unacceptable. A PCI penetration test, by definition, will be scoped toward systems and networks containing payment card data. There could be a server full of Social Security numbers nearby, but unless that system is in scope, it would not be covered (or reported on) in a PCI test report.

Scoping is important enough get its own coverage in The Penetration Testing Execution Standard, a voluntary effort backed by some well-known security professionals. The document warned that “defining scope is arguably one of the most important components of a penetration test, yet it is also one of the most overlooked.” It then provided valuable questions for the business unit managers as well as questions for the system administrators.

Getting the Most Out of a Pen Test

A Switch and Shift article offered valuable advice that can be applied to pen tests: Lessons learned are a key part of any activity. A debrief is a key tool for reviewing what worked well, identifying what didn’t and extracting the lessons learned.

The article asked a question that rings true in today’s complex security environment: “If fighter pilots and surgeons devote time to learn from their actions, how can your business afford not to do the same?” It pointed out that “a thorough debrief simply can reduce your time to insight. If done correctly, it will accelerate learning and increase experience beyond the mere time spent performing the actions being debriefed.”

A pen test is the security equivalent of running a mission. The post-test debrief is vital to getting the most out of a penetration test. What worked? What didn’t? How can we improve the next time around?

The Penetration Testing Execution Standard outlined detailed questions to ask regarding the effectiveness of the incident responders while the test was underway. Were they able to detect and respond to:

  • Information gathering;
  • Footprinting;
  • Scanning and vulnerability analyses;
  • Infiltration (attacks);
  • Data aggregation; and
  • Data exfiltration?

The takeaway here is to not let a good lesson go to waste. As you await your report, assemble your team, incident responders and those charged with decision-making or communication responsibilities to debrief the mission.

Have this initial debrief with your team while things are still top of mind. Once the report is in, merge the timelines and see where the gaps appeared. Better yet, include the pen testers in your team debrief to extract the most from your experience.

Read the interactive white paper: Preempt attacks with programmatic and active testing

More from Application Security

Patch Tuesday -> Exploit Wednesday: Pwning Windows Ancillary Function Driver for WinSock (afd.sys) in 24 Hours

‘Patch Tuesday, Exploit Wednesday’ is an old hacker adage that refers to the weaponization of vulnerabilities the day after monthly security patches become publicly available. As security improves and exploit mitigations become more sophisticated, the amount of research and development required to craft a weaponized exploit has increased. This is especially relevant for memory corruption vulnerabilities.Figure 1 — Exploitation timelineHowever, with the addition of new features (and memory-unsafe C code) in the Windows 11 kernel, ripe new attack surfaces can…

Backdoor Deployment and Ransomware: Top Threats Identified in X-Force Threat Intelligence Index 2023

Deployment of backdoors was the number one action on objective taken by threat actors last year, according to the 2023 IBM Security X-Force Threat Intelligence Index — a comprehensive analysis of our research data collected throughout the year. Backdoor access is now among the hottest commodities on the dark web and can sell for thousands of dollars, compared to credit card data — which can go for as low as $10. On the dark web — a veritable eBay for…

Direct Kernel Object Manipulation (DKOM) Attacks on ETW Providers

Overview In this post, IBM Security X-Force Red offensive hackers analyze how attackers, with elevated privileges, can use their access to stage Windows Kernel post-exploitation capabilities. Over the last few years, public accounts have increasingly shown that less sophisticated attackers are using this technique to achieve their objectives. It is therefore important that we put a spotlight on this capability and learn more about its potential impact. Specifically, in this post, we will evaluate how Kernel post-exploitation can be used…

Detecting the Undetected: The Risk to Your Info

IBM’s Advanced Threat Detection and Response Team (ATDR) has seen an increase in the malware family known as information stealers in the wild over the past year. Info stealers are malware with the capability of scanning for and exfiltrating data and credentials from your device. When executed, they begin scanning for and copying various directories that usually contain some sort of sensitive information or credentials including web and login data from Chrome, Firefox, and Microsoft Edge. In other instances, they…