The following is a response to a preview article I wrote on pentesting coverage. The person I had the IM discussion with was Daniel Miessler. He responded in his own blog, and sent me the excerpt below as a response. Thanks for the offline and online comments to far. Certainly an interesting topic to discuss!
My view is much different than Johannes's. I feel the test type he's describing is a detailed vulnerability assessment-not a penetration test. I think the key difference is opacity. He says that he doesn't believe in black box testing, and that's fine; I too agree that white box testing is far more effective.
But I think very concept of white box vs. black box is, by default, a discussion about vulnerability assessment and not penetration testing because penetration testing implies black box. In other words, you can have a white box or black box vulnerability assessment, but you can only have a black box pentest.
This is because the analog to the pentest (in my opinion) is the elite military unit commissioned to test a military base's security. See Tiger Team. Many readers may remember Richard Marcinko, from SEAL Team 6 who used to break in and kidnap Admirals and hijack nuclear subs. This is a pentest in my view-you are given a single goal by the client: "get as far as you can", or "see what you can get out of my database" or "try and modify my payroll records".
The mission of the pentest team is to achieve the goal that has been given-not to find all (or even many) vulnerabilities in the target's defenses. Any vulnerability assessment that takes place against the target will be solely for the purpose of finding a way in-nothing more. And if they get in on the first try, and accomplish the goal, then the report will indicate as much.
The report will state how they got in, what the customer should do to shore things up from that vector, and perhaps mention a few other things in passing. But it won't be a comprehensive review of the customer's security posture. That would be a separate engagement-a *vulnerability assessment* engagement.
Main Points
So here are my main propositions:
- A vulnerability assessment focuses on breadth: the goal is to identify as many issues as possible. This is why white box VAs are generally a superior option if the testing team is skilled enough to provide one.
- A penetration test focuses on depth, not breadth: the focus is on achieving a pre-determined goal that could only be possible if security were to fail, and not to find vulnerabilities. Vulnerabilities are used in a pentest, but they aren't the focus. The focus is achieving the pre-determined goal.
- Vulnerability Assessments are (or should be) requisitioned by those who already know they have many issues and simply need help identifying and prioritizing them. The more issues identified the better, so naturally a white box approach should be embraced when possible.
- Penetration Tests should only be commissioned as a final stage of a security program, by those who believe their security program is mature, functioning well, and wish to test it against an adversary. It is, by nature, black box because the goal is to simulate a real-world attacker.
Daniel Miessler's complete post on vulnerability assessments vs. penetration testing can be found here.
