Battle of the Colored Boxes (part 1 of 2)

Black, White, and Gray Box software testing methodologies, everyone has a preference. People frequently claim one method is best because another is “limited”. For myself that’s like saying a fork is better than a spoon (at what exactly?). The fact is if a perfect methodology for software testing existed, we’d have bug free code, but of course we don’t. Each has its strengths, which are often the weaknesses of others. This means the various methodologies are often complimentary to each other, not substitutes. The best thing to do is understand what they are good and not so good to select the one most appropriate for the task.

Lets look at Black, White, and Gray Box software testing from a high-level as it relates to a website security standpoint and highlight their strong points. I realize that not everyone will agree with my conclusions. So as always, feel free to comment and let me know if anything has been overlooked and should be considered. Also for perspective I’m of the opinion that all three methodologies require tools (scanners) and experienced personnel as part of the process. No exceptions.

Black Box (Dynamic Analysis)
Wikipedia: “Dynamic code analysis is the analysis of computer software that is performed with executing programs built from that software on a real or virtual processor”.

An attacker starting off with zero knowledge any application source code, system access, documentation or anything a typical user wouldn't have access to. This is normally an attacker with a web browser, proxy, and perhaps some fault injection tools at their disposal.

Black Box’ing:

1. Measures the amount of effort required for an attacker to compromise the data on a website. This measurement takes into consideration additional layers of defense (web servers, WAFs, permissions, configs, proxies, etc.). This enables website owners to focus on areas that directly improve security.
2. Allows faster business logic testing by leveraging actual operational context. Testers are able to uncover flaws (Information Leakage, Insufficient Authorization, Insufficient Authentication, etc.) that may otherwise not be visible (or significantly harder to find) by analyzing vast amounts of source code.
3. Is generally considered to be faster, more repeatable, and less expensive than White Box’ing. This is helpful to development environments where websites are updated more than a few times per year and as a result requires constant security (re)-assurance.
4. Provides coverage for common vulnerabilities that are not present in the code, such as Predictable Resource Location and Information Leakage. Files containing sensitive information (payment logs, backups, debug messages, source code, etc.) routinely become unlinked, orphaned, and forgotten.

White Box (Static Analysis)
Wikipedia: “Static code analysis is the analysis of computer software that is performed without actually executing programs built from that software. In most cases the analysis is performed on some version of the source code and in the other cases some form of the object code.”

An attacker with access to design document, source code, and other internal system information.

White Box’ing:

1. Generally considered to a be a deeper method of software testing as it can touch more of the code that may be very difficult or impossible to access from the outside. Vulnerabilities such as SQL Injection, backdoors, buffer overflows, and privacy violations become easier to identify and determine if an exploit will work.
2. Can be employed much earlier in the software development lifecycle (SDLC) because Black Box’ing requires that websites and applications are at least somewhat operational. Libraries and APIs can be tested early and independently of the rest of the system.
3. Is capable of recommending secure coding best-practices and pinpoint the exact file or line number of vulnerabilities. While a website might be “vulnerability free” from an external perspective, bad design decisions may cause a precarious security posture. Weak use of encryption, insufficient logging, and insecure data storage are examples of issues that often prove problematic.
4. Has an easier time understanding if identified vulnerabilities are of a one-off developer mistakes or architectural problems across the entire application infrastructure. This insight in useful when strategizing security decisions such as training or standardizing on software frameworks and APIs or both.

Gray Box or Glass Box
The combination of both Black and White Box methodologies. The spork of software testing if you will. This approach takes all the capabilities of both and reduces their respective drawbacks. The goal is to make the whole (process) worth more than the sum of its parts. In many ways Gray Box’ing achieves this so no need to go back over the above material. The largest negative issue is the increased cost in time, skill, repeatability, and overall expense of the process. Qualified people proficient in either Black or White Box’ing are hard to find and retain. Locating someone who is solid at both is extremely rare and as a result they can demand high bill rates. And of course they need double the tools so double the cost.