Where's WhiteHat? Re: Scanner Comparisons

Last week Larry Suto published, “Analyzing the Accuracy and Time Costs of Web Application Security Scanners,” which reviewed desktop black box website vulnerability scanners: Acunetix, IBM AppScan, BurpSuitePro, Cenzic Hailstorm, HP WebInspect, NTOSpider, and Qualys WAS (Software-as-a-Service). This research was meant to build upon Larry’s initial October 2007 study of the market. Several people have asked what I thought of the paper and why WhiteHat Security wasn’t covered. I’m happy to discuss both questions.

First off, the website vulnerability management space is in desperate need of side-by-side comparisons as they are very few and far between. This represents a challenge for organizations building application security programs that want to create a product short-list to evaluate internally. For a variety of reasons, independent and publicly available detailed technical reviews of website vulnerability management products/solutions are unlikely to come from the expected sources, such trade magazines and industry analysts. One of the main inhibitors is that many of these firms do not have a Web application testbed that would allow for an accurate, fair comparison. Thankfully, researchers, such as Larry Suto, help fill the void by investing personal time and sharing their results. Others independents should be encouraged to do the same. I would recommend the “Web Application Security Scanner Evaluation Criteria (WASSEC)” if you wish to do so. Also, Larry confirmed he was not compensated by any vendor for his work.

During the latter stages of Larry’s research, WhiteHat Security was offered the opportunity to be included. In order to take part, we were asked to assess six vendor hosted “test websites” with WhiteHat Sentinel, identify vulnerabilities and report our findings. The testing process was designed to evaluate desktop black box scanner technology using a “Point-and-Shoot” and “Trained” methodology -- something not well-suited for a SaaS offering like Sentinel. What we do and how we do it is very different from the scanning tools, but is still often seen as an alternative. After much consideration, we did not feel that the testing methodology would provide an apples-to-apples evaluation environment for Sentinel. Additionally, and equally important, finding vulnerabilities is only the first piece of the overall value our customers receive, so we politely declined to participate.

*Not to mention, that as a strict rule, we never touch any website without express written authorization.*

The report states, consistent with my expectations, that significant amounts of human/expert time are required for scanner configuration, vulnerability verification, scan monitoring, etc. to enable the aforementioned products to function proficiently. Behind the scenes, Sentinel is no different, except that we perform all of those activities and more for our customers as a standard part of the annual subscription. Doing so means our engineering time has a hard cost to us and staff resources are dedicated to serving customers.

The report confirms that scanner performance varies wildly from site to site, so it’s best to test where they will be deployed. We agree and provide mechanisms for our customers to evaluate Sentinel first-hand for exactly that reason. We want customers to know exactly what they can expect from us on their production systems and not a generic test website. Finally, where Sentinel really excels, is in areas where the report did not (could not) focus. Scalability is one of these. Whether we are tasked with one site, 10, 100, 1,000 or more -- Sentinel is capable of handling the volume.

Over the years I’ve written extensively about black box scanner efficiencies and deficiencies in depth: Automated Scanners vs. Low-Hanging Fruit, Automated Scanner vs. The OWASP Top Ten, shed light on duplication rates, what scanners are good at finding and what they are not -- with a scan-o-meter, discussed business logic flaws, why crawling matters a lot, how much essential human time is required, and even what it takes to make the best scanner. So obviously the scanners overall general poor showing came as no surprise. Most missed nearly half of the vulnerabilities, on test websites no less, where they are designed to be easily found. Imagine how they would perform in the real world! It gets much uglier and dangerous out there I assure you.

I’d highly recommended everyone interested Web application security read Larry’s report for themselves and get familiar with what is the current state-of-the-art in black box scanning products. Much improvement could be made through additional R&D. Remember, keep an open mind, but at the same time take the results with a grain of salt until you test on your systems. Some vendors will say Larry’s work wasn’t perfectly scientific, possessed data errors, was misinterpreted, ran misconfigured, couldn’t reproduce the results, etc. All of that may be true, but so is the fact that it looks like Larry conducted a deeper and fairer analysis than the average would-be user does during typical evaluations.

Here are the three takeaways:

1. Scanner results will vary wildly from website to website. For best results, I highly recommend that you evaluate them on the sites where they are going to be deployed. Better still, if you know of vulnerabilities in those sites ahead of time, use that information for comparison purposes.
2. The significant human/expert time required for proper scanner set-up, ongoing configuration, vulnerability verification, etc. must be taken into consideration. Which vulnerabilities you scan for is just as important as how you scan for them. Estimate about 20-40 man-hours per website scanned.
3. Scanner vendors should take into consideration that Larry Suto is certainly more sophisticated than the average user. So if he couldn’t figure out how to run your tool “properly,” take that as constructive feedback.