Wednesday, November 22, 2006

We don' know what we don't know

I’m frequently asked about “completeness” when it comes to vulnerability scanning/assessments. “How do you know if all the vulnerabilities have been found?” The short answer is you don’t. Then I proceed to describe the reasons. What got me thinking was why this question is asked so often. I think the answer is only obvious to those experienced in web application security. These are faulty assumptions routinely carried over from network vulnerability scanning world that do not apply to webappsec.

In network scanning the list of “well-known” vulnerabilities is large, but also finite. Databases such as OSVDB, SecurityFocus, MITRE (CVE), and others catalog the known universe of issues. Vulnerability coverage by network scanners is likely close to 100%. In “custom” web applications the luxury of well-known vulnerabilities or database repositories vanishes. Each new vulnerability identified is more or less a one-off / zero-day issue. Just as with bugs in application code, we truly never know how many vulnerabilities exist in a web bank, e-commerce store, payroll system, or any other custom web application. The upper bound in an unknown. Therefore we can never know for sure if any scan/assessment found them all. Vulnerability coverage could be as low as 10-20% or higher in the range of 80-90% or more. The point is we don’t know, its difficult to measure, and changes with each website.

This is a big reason why I’ve been talking a lot about measuring security recently. I’m a big believer in it. Who isn’t? I even took a shot at a Methodology for Comparing Web Application Vulnerability Assessment Solutions. Figured we could use time-it-takes-to-hack-a-website as something we could reliably measure. For some reason I hadn’t got much feedback on the idea. Likely because there hasn’t been customer demand as they’re not REALLY aware of the fact that everything isn’t being found. Whether my methodology works or not, we’re going to need to figure this out. Once customers of ANY webappsec VA solutions gets hacked due to missed vulnerabilities, there’s going to be hell to pay.


Anonymous said...

Excellent graphic. Although there is no master agreed-upon list of vulns in the app space, surely there are buckets that we can (and do) divide the unknown into?

Jeremiah Grossman said...

Heya Davi!

> Excellent graphic.

Trying to flex my meager creative talents.

> Although there is no master agreed-upon list of vulns in the app space, surely there are buckets that we can (and do) divide the unknown into?

Scan/Assessment methodologies will look for all the XSS/SQLInj/BizLogic/etc vulnerabilities it can find. After we find a vuln, we can toss it into the well-known and agreed upon buckets. What we don't know is how many of these issues are actually present in a given custom web application.

That's the problem. We can always measure who finds more than the next guy, but its hard to measure against the unknown overall vuln total, which really matters most.