Update: 10.23.2007 - Wow, people really seem to get emotional about measuring scanner effectiveness. I thought the war of words was limited to the comments on mine and RSnake's blog, but apparently NTOBJECTives came under DDoS attack after Larry's review was released. Looks like it died down cause I get to their website.
Looks like this...
Within a few moments of pressing the scan button it’ll find every vulnerability, with zero false positives, generate a pretty looking report, and voila you’re compliant with GLBA, HIPAA, and PCI-DSS. Of course, we all know such a web application scanner is simply not possible to create for a variety of reasons. That’s why each feature or configuration option in the GUI should be considered, compensating for a technical limitation. For example, if ScannerA has a feature ScannerB lacks, this doesn’t mean ScannerB is missing something. It could very well mean ScannerB overcame a hurdle that ScannerA still needs a human to complete. Or, maybe it means ScannerB is indeed limited. It’s often hard to tell which is which even for an expert.
As another example, some scanners have a GUI option to configure what a customized 404 Not Found page looks like. Others don’t need any assistance, because an algorithm handles that logic automatically. Some scanners offer both options just in case. There are many similar examples. Until the scanner is run on a target website, it is impossible to tell what the outcome will be ahead of time; and, even after the fact it’s still tricky to figure out what happened. As such, scanners are not designed to perform an entire vulnerability assessment on their own. At least I hope not. Scanners are designed to help a person save time in completing the process. Unless this is explained up front to customers, they’ll have improperly set expectations and eventual disappointment.
This brings us to the only two reviews publish this year by Jordan Wiens of Network Computing and Larry Suto (Application Security Consultant in San Francisco). Jordan’s Web Application Scanners Rolling Reviews focused on Ajax vulnerability detection with proper tool configuration, while Larry analyzed coverage depth in plain ol’ default scan mode. Guess what!? Very different rankings occurred amongst the scanners. In Network Computing, though the scanners tested (WebInspect, Acunetix, Hailstorm, and N-Stalker) claimed capability of supporting Ajax automatically, all failed except AppScan. Then Larry’s results had little known NTOSpider with top honors for ability to scan deeper than both AppScan and WebInspect. Strange eh? I highly recommend the reading both reviews and drawing your own conclusions.
Most disappointingly though for WhiteHat Security, Jordan ran out of time on his project before he was able to write up a full review of our Sentinel offering. I was really excited about the opportunity to demonstrate how our SaaS technology could spank all the scanners in Ajax support, vulnerability identification, false-positive rate, ease of deployment, reporting, ROI, and any other metric that matters. Another time and another place I guess. However, we were able to run Sentinel through the same environment as everyone else and generate results. So, Jordan was able to publish the following kind words in his follow up that we appreciated. It speaks for itself:
“Besides nabbing all the vulnerabilities discovered by the scanning products, WhiteHat's Sentinel identified e-mail-based XSS vulnerabilities in our sample Web mail application through its combination of manual testing and automated scanning. WhiteHat navigated all our sample Ajax applications without any trouble.
Based on our testing, if you want automated scans of Ajax applications, your best options are Sentinel and AppScan.”
Here’s what I want to leave the post with. Evaluating web application vulnerability scanners is a difficult task for anyone. A person has to be knowledgeable in web application security, capable of understanding the report results, not to mention be able to set up enough real-world websites to make the comparison reasonable. How many people does that eliminate? Then what to measure? Everyone has a different point of view of what is meaningful. Do we measure vuln to vulns, Ajax support, scanner depth, usability, reporting capabilities, etc? Each metric has value, but not to everyone all the time. To assist, Anurag Agarwal is helping WASC create a Web Application Security Scanner Evaluation Criteria (WASSEC) with assistance from the community. It should be highly useful when completed.