The new OWASP Top 10 2007 has recently be made available. Excellent work on behalf of all the contributors. As described on the website, “This document is first and foremost an education piece, not a standard.”, and it’ll do just that. Educate. Last week I provided project team with updated text (unpublished) that more accurately describes the current capabilities of “black box” automated scanners in identifying the various issues on the list. The exercise provided ideas for the remainder of this blog post; estimating how effective scanners are at finding the issues organized by OWASP Top-10.
In the past I’ve covered the challenges of automated scanning from a variety of angles including technical vs. logical, vs. low hanging fruit, vs. the OWASP Top 10, and in one occasion I threw down the gauntlet. Ory Segal (Director of Security Research, Watchfire) also weighed in with his thoughts via his shiny new blog that’s worth a read. Everyone agrees scanners find vulnerabilities, though most including product vendors admit they certainly don’t find everything. But that doesn’t explain nearly enough. Does this mean scanners find almost everything or just above some? Where does the needle land on the web application scan-o-meter? Sure, scanners are good at finding XSS. How good? Scanners are bad at identifying flaws in business logic. How bad? This is a very limited understanding and we could really use more insight into scanner capabilities with quantification of capacity.
Going by experience in developing scanner technology for the better part of a decade, testing scanners written by many others, feedback from the surveys, personal conversation, and quality time huddled with Arian Evans (Director of Operations, WhiteHat Security) — below are estimates of where I think state-of-art scanning technology has reached when speaking of relative averages. This is actually a harder exercise than you might think when all things are considered. I invite others to comment from their experience as well whether they agree or disagree. Should any of the scan-o-meter readings be higher or lower or anything I might not have considered?
Estimates are based on the completed automated results of a well-configured scanner, that gets a good website/web application crawl and is able to maintain login state during testing. We’ll also focus only on vulnerabilities that are exploitable by a remote-external attacker. In short, a best chance/case scenario.
A1 - Cross Site Scripting (XSS)
A2 - Injection Flaws
A3 - Malicious File Execution
A4 - Insecure Direct Object Reference
A5 - Cross Site Request Forgery (CSRF)
Identification of CSRF turns out to be fairly easy, filtering out which issues we care about is where automation falls down. That’s what this meter reflects, purely automated results.
A6 - Information Leakage and Improper Error Handling
A7 - Broken Authentication and Session Management
A8 - Insecure Cryptographic Storage
A9 - Insecure Communications
A scanners challenge with this issue is putting it in terms of business expectations. Perhaps a website does not need or want SSL is certain places and that’s OK.
A10 - Failure to Restrict URL Access