PCI Meetup #8: When is a PCI Penetration Test Good Enough?

In the latest PCI Meetup, the sole agenda was again the PCI penetration test, and pentest quality/standards in general. In my last post, I pointed out a few existing standards and some that are currently in development. The general consensus in the meetup is that existing standards are insufficient, and that any hope of creating a baseline for pentest quality rests on future efforts to draft an industry standard.

For a moment, let us hypothesize that a robust pentest standard has been created, and the PCI council is willing to incorporate it into the PCI DSS. What else do we need? With any standard, we will still need parameters specific to PCI for it to be useful. With the wrong parameters, the very best standard won't achieve our goal of determining whether or not cardholder data is reasonably safe, and we can't assume the pentester will assume the correct parameters. In addition to requiring a penetration testing standard, the PCI DSS should also detail how the test should be executed, in order to maximize effectiveness and ensure the test represents real-world scenarios as closely as possible.

• Scope: The PCI scope defines an island, or islands within the enterprise where cardholder data is (allegedly) protected by the PCI DSS requirements, and is segregated from the rest of the network. The penetration test should not be limited to the PCI scope. In fact, one of the goals of the penetration test should be to test the effectiveness of cardholder DMZs from the outside, and determine whether weaknesses outside the cardholder network can be used to penetrate it.
• Goals: Obtaining cardholder data should be the primary goal. Testing the effectiveness of segregation and attempting to penetrate the cardholder network defenses should also be a primary goal.
• Level/Strength of the Test: It is unreasonable to expect the basic, annual PCI penetration test to include custom, unreleased 0-day exploits. It just isn't feasible, given the cost and the number of pentesting companies/individuals capable of providing this level of testing. The penetration test should represent the skill of a fairly technical attacker with commonly available tools (e.g. Metasploit). Ideally, the penetration testing standard would use a tiered approach, similar to the OWASP Web App Standard mentioned in my last post. The PCI council could then choose the minimum tier, or baseline for merchants to be tested against.
• Attack vectors: Currently, PCI requires the penetration test to include internal, external and webapp tests. This is not sufficient. In the future, social engineering must be a requirement, and for good reason as this example shows: An administrator recently gave out the root password for rootkit.com, allegedly to a teenager. If a teenager could similarly penetrate your cardholder environment with a spoofed email or phone call, the value of the $4.5 million you spent on PCI compliance goes to $0. Instantly.

I look forward to your responses, opinions and the next meetup!

First Post

I've thought about blogging my thoughts for some time. Shortly after DefCon 18, I began organizing PCI Meetups so that some of us could continue PCI and other security-related conversations without needing to travel to an InfoSec conference to do it. I have decided the ideas coming out of these discussions needed a more public, permanent place to live.

Though I have been heavily involved with PCI since it was VISA CISP, I'm a pentester, hacker, and security enthusiast at heart. Most of the posts here will probably be PCI-related, but I've chosen the title of this blog more or less to delude myself otherwise.

PCI Meetup #7: Penetration Test Quality

We had an interesting group for the last PCI Meetup . The majority of individuals participating in the conversation were professional penetration testers. Netragard's PCI Compliance challenge sparked some good discussion on Twitter, and ended up driving the agenda for the seventh PCI Meetup in the same direction. We spent the majority of the Meetup discussing the need for a penetration test standard, especially in the PCI DSS.

Illustrating the Need

Netragard uses an analogy, pointing out that the quality of some penetration tests is akin to testing "the effectiveness of a bullet-proof vest with a squirt gun". I'm sure they're quite skilled, as no security services company would issue such a challenge unless they are:

1. Currently occupying a place of honor here, or

2. Very good at what they do.

That got me thinking though - if company A's pentest is equivalent to a squirt gun, how do I know how Netragard, or any other pentesting company for that matter, stack up against their competition? To continue the analogy, are company B's efforts equivalent to a pressure washer? A 9mm handgun? A rocket launcher? Let us examine the problem from a different point-of-view with two hypothetical proposals.

	Company A	Company B
Work Description:	Int/Ext Pentest	Int/Ext Pentest
Hours:	40	40
Price:	$8000	$8000

In this example, it appears that both companies are offering to perform the same work for the same price, with the same level of effort. Company A, however, is handing off a glorified vulnerability scan as a penetration test, whereas Company B is fuzzing and coding custom exploits as they perform what most in the industry would consider to be an A-rate penetration test. How can a customer tell the difference? Those of us in the business might suggest requesting a list of pentester credentials from each company, or a high-level synopsis of their internal pentesting methodology. Both of these are great, but they get us no closer to being able to measure Company A or Company B's ability to perform a quality penetration test.

This line of thinking brought me to two conclusions. First, we need some sort of standard, or set of standards to measure penetration test quality against, and second, that PCI should require a minimum standard for the required annual penetration test. Currently, what passes for a penetration test in PCI is entirely up to the QSA. The QSA could let Company A's pentest pass as sufficient, or they could go to the other extreme and require something that far exceeds the intent of the requirement. Without a standard, anything could conceivably be passed off as a penetration test - thus Netragard's desire to challenge the status quo.

Consider also that not all PCI requirements are equal. The PCI requirement for an annual penetration test is just one of over 200 requirements spread across 12 categories, sure, but is not on the same level of importance as say, ensuring network diagrams are up-to-date. It is, in fact, the only PCI requirement that can actually test whether or not the other requirements are likely to succeed in preventing a potential breach! The standard of quality this single requirement is held to stands between determining the effectiveness of compliance work performed, and having no clue.

The PCI DSS was likely designed so that the annual penetration test would act as a built-in test for effectiveness. If some sort of standard for quality can be assured, then PCI in general could become significantly more effective.

Standards

I realized that I am far from the first person to recognize the need for penetration testing standards. After a bit of research, I realized there was more out there than I realized.

• ISECOM's OSSTMM manual is a well-known set of security testing guidelines.
• OWASP maintains a guide for testing web-based applications and a multi-level security standard.
• There appears to be an ANSI standard now for penetration testing specifically within the financial services industry (I have not yet forked out $100 to check it out though - let me know if you have, and what your impressions are)
• The NBISE is gathering information to put together a penetration testing standard
• ...and just last night, I listened to Chris Nickerson discussing this very problem on the InfoSec Daily Podcast, and it seems he is collaborating an industry solution as well (bookmark and check back later)

I think all of these have potential to help weed out the "fakers" in the industry, but should one of these be included as part of PCI's annual penetration test requirement, or does the PCI data security standard need its own custom penetration test standard?

Conclusion

I've gone as far in the Meetups as to suggest that a good quality pentest could replace the entire PCI DSS. If the pentester gets their hands on cardholder data, you fail. If they don't, you're compliant. Is that not the bottom line? The pentest becomes, at least, a partial measure of the likelihood that a breach may occur at a given company. This has many benefits, like protecting businesses who are adequately protecting cardholder data from wasting millions unnecessarily. The major problem is that everything rests on the quality of that single annual penetration test.

If there is enough interest, we may discuss this again in the eighth PCI Meetup, which will take place this Thursday, February 10th, 2011 at 8PM EST. If you want to join the discussion, drop me a note here or on Twitter with your SkypeID, and I'll make sure to include you in the discussion!