As a penetration tester, you will inevitably find yourself involved in scoping engagements, navigating the challenges of aligning a client’s needs with their expectations. “Penetration testing” is a term that almost everyone believes they understand, yet it often carries vastly different interpretations. This ambiguity makes it more of an umbrella term, offering little insight into how the test will actually be executed or what it will cover. In this post, I’ll share my perspective on one approach to differentiating how penetration tests can be executed, helping both testers and clients clarify their expectations.

When discussing penetration testing, it’s helpful to break down the ways a test can be executed into distinct approaches. This is just one of many ways to clarify expectations and align the scope of a test with the goals of both the tester and the client. In this post, I’ll focus on three key areas, each of which requires a different set of skills and expertise:

  1. Compliance testing involves executing a predetermined list of test cases to ensure the system adheres to specific standards or regulatory requirements. This type of testing benefits from familiarity with the relevant compliance frameworks and the ability to systematically assess against defined criteria.
  2. Configuration weaknesses and known vulnerabilities focuses on identifying and exploiting known vulnerabilities and misconfigurations within the system’s components, leveraging established methods to evaluate the system’s current posture. Expertise in vulnerability databases, common misconfigurations, and practical exploitation techniques is critical here.
  3. In vulnerability research the goal is to discover zero-day vulnerabilities—previously unknown flaws within the system—through deeper analysis. This requires advanced skills in reverse engineering, protocol analysis, and often a creative mindset to uncover new attack vectors.

Each of these approaches offers valuable insights, but understanding their differences is essential for setting clear expectations and aligning the test to the organization’s needs.

Compliance Testing

Compliance testing is valued for its repeatability. By following a predetermined set of test cases, it ensures consistency across tests, making it easier to measure improvements over time and identify recurring issues. This makes it an excellent choice for organizations with strict standards or regulatory requirements.

However, the structured nature of compliance testing can also be a limitation. It lacks the exploratory element that allows testers to uncover novel or unexpected vulnerabilities, as it focuses strictly on predefined test cases. Despite this, compliance testing can be particularly useful in rapidly changing environments, as it ensures consistent coverage with every iteration.

Frameworks like PCI-DSS, focused on securing payment card data, and OWASP ASVS, which sets security standards for web applications, provide clear guidelines for compliance testing. While this approach may not catch everything, it is essential for organizations to demonstrate adherence to industry standards, build trust, and maintain accountability.

Configuration Weaknesses and Known Vulnerabilities

This type of testing often covers a broad scope, such as an entire organization or a data center. Unlike compliance testing, it is typically black or gray box, focusing on what can be observed or accessed from a specified starting point. The goal is to identify and exploit known vulnerabilities or misconfigurations to see how far an attacker could progress towards a predetermined objective.

This approach is particularly valuable for assessing the organization’s current security posture against well-documented risks. Testers rely on vulnerability databases, exploit frameworks, and knowledge of common misconfigurations to simulate realistic attack scenarios. For example, they might exploit unpatched software, default credentials, or exposed services to gain deeper access into the environment.

The advantage of this method is its ability to uncover weaknesses that arise from day-to-day operations or overlooked configurations. However, it is not designed to find new, undiscovered vulnerabilities, as its focus remains on existing attack vectors. Organizations benefit from this type of testing by identifying practical areas for improvement, such as patch management, network segmentation, and access controls.

Vulnerability Research

Vulnerability research is typically narrower in scope, focusing on a single system, application, or specific component. Unlike other types of testing, its primary goal is to uncover unknown vulnerabilities—commonly referred to as zero-days—that have not yet been documented or exploited. This makes it one of the most resource-intensive approaches to penetration testing.

The process often involves deep analysis of the target system, including reverse engineering, protocol analysis, and code review. Testers employ creative problem-solving and advanced techniques to identify weaknesses that standard testing methods might miss. For example, they might analyze how an application processes input, searching for subtle errors that could lead to memory corruption or privilege escalation.

This type of testing is particularly important for vendors and organizations managing critical systems. For vendors, performing even basic vulnerability research during development helps identify and address zero-day vulnerabilities before release, reducing the risk of shipping insecure software. For critical systems or those likely to face targeted attacks, discovering even a single zero-day can have significant implications, making this approach invaluable in safeguarding users and maintaining trust.

Conclusion

Penetration testing is not a one-size-fits-all process. Each of the approaches discussed—compliance testing, configuration weaknesses and known vulnerabilities, and vulnerability research—serves a unique purpose, addressing different aspects of an organization’s security. Compliance testing ensures consistency and alignment with standards, configuration testing identifies practical vulnerabilities within a broader scope, and vulnerability research uncovers critical zero-days within specific systems.

Choosing the right type of penetration test depends on your goals, resources, and the systems in scope. Equally important is finding the correct expertise for the job. Each approach requires specialized skills, from regulatory knowledge for compliance testing to deep technical expertise for vulnerability research. The success of a penetration test lies not only in the methodology but also in the proficiency of the testers conducting it.

By understanding the differences and value of each approach, organizations can better align their testing strategies with their needs, ultimately strengthening their overall security posture.