“Experience is a hard teacher because she gives the test first, the lesson afterwards.” – Vern Law

The penetration test is an important tool to help organizations assess the state of their defenses. The National Institute of Standards and Technology (NIST) defined a pen test as “a test methodology in which assessors, using all available documentation (e.g., system design, source code, manuals) and working under specific constraints, attempt to circumvent the security features of an information system.”

Why Are Penetration Tests Necessary?

While it may be tempting for business executives to look at a pen test as a hard measure of the organization’s security defenses, not all pen tests are created equal. Even NIST warned that “penetration testing represents the results of a specific assessor or group of assessors at a specific point in time using agreed-upon rules of engagement.”

Much depends on how the pen test is scoped. NIST cautioned that “considering the complexity of the information technologies commonly employed by organizations today, penetration testing can be viewed not as a means to verify the security or privacy features of an information system, but rather as a means to: (i) enhance the organization’s understanding of the system; (ii) uncover weaknesses or deficiencies in the system; and (iii) indicate the level of effort required on the part of adversaries to breach the system safeguards.”

Compared to a vulnerability scan, a penetration test provides “an explicit and often dramatic proof of mission risks and an indicator of the level of effort an adversary would need to expend in order to cause harm to the organization’s operations and assets, to individuals, to other organizations.”

However, the test also provides something that is often missing when reviewing the effectiveness of security spending: an assessment of how well the organization is able to respond to an attacker — albeit an attacker whose rules of engagement are well-defined.

Read the interactive white paper: Preempt attacks with programmatic and active testing

Scoping Is Key

Determining the correct scope and parameters is key to ensuring the organization gets good value out of this expenditure. An overly broad pen test will cost more and is likely to disrupt business systems, as well as other systems that may not house any sensitive or valuable data. On the opposite end of the spectrum, a narrowly scoped test may only focus on a few key systems and miss other relevant systems that house critical data (e.g., operations data, intellectual property data, HR data, HIPAA data and payroll data).

In its Penetration Testing Guidance report, the PCI Security Standards Council cautioned that the organization should work closely with the pen tester during scoping “to verify that no components are overlooked and to determine whether any additional systems should be included in scope. The scope of the penetration test should be representative of all access points, critical systems and segmentation methodologies” for the system(s) covered.

While 10 years ago it might have been acceptable for a business executive to mistake a successful PCI pen test to mean that all the organization’s systems were safe, doing so in 2016 is unacceptable. A PCI penetration test, by definition, will be scoped toward systems and networks containing payment card data. There could be a server full of Social Security numbers nearby, but unless that system is in scope, it would not be covered (or reported on) in a PCI test report.

Scoping is important enough get its own coverage in The Penetration Testing Execution Standard, a voluntary effort backed by some well-known security professionals. The document warned that “defining scope is arguably one of the most important components of a penetration test, yet it is also one of the most overlooked.” It then provided valuable questions for the business unit managers as well as questions for the system administrators.

Getting the Most Out of a Pen Test

A Switch and Shift article offered valuable advice that can be applied to pen tests: Lessons learned are a key part of any activity. A debrief is a key tool for reviewing what worked well, identifying what didn’t and extracting the lessons learned.

The article asked a question that rings true in today’s complex security environment: “If fighter pilots and surgeons devote time to learn from their actions, how can your business afford not to do the same?” It pointed out that “a thorough debrief simply can reduce your time to insight. If done correctly, it will accelerate learning and increase experience beyond the mere time spent performing the actions being debriefed.”

A pen test is the security equivalent of running a mission. The post-test debrief is vital to getting the most out of a penetration test. What worked? What didn’t? How can we improve the next time around?

The Penetration Testing Execution Standard outlined detailed questions to ask regarding the effectiveness of the incident responders while the test was underway. Were they able to detect and respond to:

• Information gathering;
• Footprinting;
• Scanning and vulnerability analyses;
• Infiltration (attacks);
• Data aggregation; and
• Data exfiltration?

The takeaway here is to not let a good lesson go to waste. As you await your report, assemble your team, incident responders and those charged with decision-making or communication responsibilities to debrief the mission.

Have this initial debrief with your team while things are still top of mind. Once the report is in, merge the timelines and see where the gaps appeared. Better yet, include the pen testers in your team debrief to extract the most from your experience.

Read the interactive white paper: Preempt attacks with programmatic and active testing