PCI Meetup #7: Penetration Test Quality

We had an interesting group for the last PCI Meetup . The majority of individuals participating in the conversation were professional penetration testers. Netragard's PCI Compliance challenge sparked some good discussion on Twitter, and ended up driving the agenda for the seventh PCI Meetup in the same direction. We spent the majority of the Meetup discussing the need for a penetration test standard, especially in the PCI DSS.

Illustrating the Need

Netragard uses an analogy, pointing out that the quality of some penetration tests is akin to testing "the effectiveness of a bullet-proof vest with a squirt gun". I'm sure they're quite skilled, as no security services company would issue such a challenge unless they are:

1. Currently occupying a place of honor here, or

2. Very good at what they do.

That got me thinking though - if company A's pentest is equivalent to a squirt gun, how do I know how Netragard, or any other pentesting company for that matter, stack up against their competition? To continue the analogy, are company B's efforts equivalent to a pressure washer? A 9mm handgun? A rocket launcher? Let us examine the problem from a different point-of-view with two hypothetical proposals.

	Company A	Company B
Work Description:	Int/Ext Pentest	Int/Ext Pentest
Hours:	40	40
Price:	$8000	$8000

In this example, it appears that both companies are offering to perform the same work for the same price, with the same level of effort. Company A, however, is handing off a glorified vulnerability scan as a penetration test, whereas Company B is fuzzing and coding custom exploits as they perform what most in the industry would consider to be an A-rate penetration test. How can a customer tell the difference? Those of us in the business might suggest requesting a list of pentester credentials from each company, or a high-level synopsis of their internal pentesting methodology. Both of these are great, but they get us no closer to being able to measure Company A or Company B's ability to perform a quality penetration test.

This line of thinking brought me to two conclusions. First, we need some sort of standard, or set of standards to measure penetration test quality against, and second, that PCI should require a minimum standard for the required annual penetration test. Currently, what passes for a penetration test in PCI is entirely up to the QSA. The QSA could let Company A's pentest pass as sufficient, or they could go to the other extreme and require something that far exceeds the intent of the requirement. Without a standard, anything could conceivably be passed off as a penetration test - thus Netragard's desire to challenge the status quo.

Consider also that not all PCI requirements are equal. The PCI requirement for an annual penetration test is just one of over 200 requirements spread across 12 categories, sure, but is not on the same level of importance as say, ensuring network diagrams are up-to-date. It is, in fact, the only PCI requirement that can actually test whether or not the other requirements are likely to succeed in preventing a potential breach! The standard of quality this single requirement is held to stands between determining the effectiveness of compliance work performed, and having no clue.

The PCI DSS was likely designed so that the annual penetration test would act as a built-in test for effectiveness. If some sort of standard for quality can be assured, then PCI in general could become significantly more effective.

Standards

I realized that I am far from the first person to recognize the need for penetration testing standards. After a bit of research, I realized there was more out there than I realized.

• ISECOM's OSSTMM manual is a well-known set of security testing guidelines.
• OWASP maintains a guide for testing web-based applications and a multi-level security standard.
• There appears to be an ANSI standard now for penetration testing specifically within the financial services industry (I have not yet forked out $100 to check it out though - let me know if you have, and what your impressions are)
• The NBISE is gathering information to put together a penetration testing standard
• ...and just last night, I listened to Chris Nickerson discussing this very problem on the InfoSec Daily Podcast, and it seems he is collaborating an industry solution as well (bookmark and check back later)

I think all of these have potential to help weed out the "fakers" in the industry, but should one of these be included as part of PCI's annual penetration test requirement, or does the PCI data security standard need its own custom penetration test standard?

Conclusion

I've gone as far in the Meetups as to suggest that a good quality pentest could replace the entire PCI DSS. If the pentester gets their hands on cardholder data, you fail. If they don't, you're compliant. Is that not the bottom line? The pentest becomes, at least, a partial measure of the likelihood that a breach may occur at a given company. This has many benefits, like protecting businesses who are adequately protecting cardholder data from wasting millions unnecessarily. The major problem is that everything rests on the quality of that single annual penetration test.

If there is enough interest, we may discuss this again in the eighth PCI Meetup, which will take place this Thursday, February 10th, 2011 at 8PM EST. If you want to join the discussion, drop me a note here or on Twitter with your SkypeID, and I'll make sure to include you in the discussion!