Methodology for Comparing Web Application Vulnerability Assessment Solutions

Measuring the time difference for a web application security expert to hack a website before and after implementing a WAVA solution.

Priority one in web application security is ensuring websites do not get hacked. Everything we do security-wise should be designed to meet this goal. If vulnerabilities exist in websites we’re responsible for, we need to find and fix them quick before ending up on the front page of Slashdot or the Wall Street Journal, on full-disclosure’s or sla.ckers.org’s wall-of-shame, or much worse by a call from the FTC. Scanners, assessments, and managed services are the 3 options organizations have when shopping for solutions to identify web application vulnerabilities. The challenge we have in the industry is how to go about usefully comparing Web Application Vulnerability Assessment (WAVA) solutions.

No website is 100% secure, at least not all the time, but there are ways to measure its security resilience and the improvement over time. This capability can be used to compare the effectiveness of WAVA solutions. WAVA solutions improve security by identifying vulnerabilities so they can be resolved before being exploited. Using this data we can begin answering the question, “how hard does the WAVA solution make websites to hack”? As such the more time, effort, and skill required to hack a website after implementing a given solution the more effective it is. The premise being is that it only takes the use of a single vulnerability to compromise a website and defraud its users.

With feedback from the community I’m hoping to improve upon the methodology to make it useful for best-practices, enterprise bake-off’s, magazine reviews, analyst reports, consultants advice, etc. Thanks to Richard Bejtlich (TaoSecurity), Robert Martin and Christy (MITRE), and RSnake for their assistance in vetting these ideas.

Definitions
o Hacker: Web application security experts with at least two years of experience in identifying and exploiting vulnerabilities.
o Hack/Vulnerability: Exploitable web application vulnerability (WASC Threat Classification) of Level-3 severity or greater under the Payment Card Industry (PCI) Severity Rating Chart.
o Test Website: A website with at least 5 or more vulnerabilities.

Assumptions
o It only takes the use of a single vulnerability to compromise a website and defraud its users.
o If the website requires login, the hacker is provided at least one test account.
o Hacker may use any tools or information gathering resources at their disposal (scanners, proxies, browser, Google, etc.).

Procedure
(Repeat procedure for 6 hackers on 6 different test websites while alternating the WAVA solution for each website)

Step #1
Hacker attempts to find a vulnerability in the website. Measure the amount of time it took for identification. (T1)

Step #2
Run the WAVA solution to identify vulnerabilities.

Step #3
Note whether or not the solution identified hacker found vulnerability from Step #1. (V1)

Step #4
Resolve all identified vulnerabilities.

Step #5
Hacker attempts to find a vulnerability in a website. Measure the amount of time it took for identification. (T2)


Data Chart

Deliverables
o All raw data charts.
o Average years of experience for the hackers.
o Average T1 time.
o Percentage of time the WAVA solution identified V1.
o Average delta between T1 and T2 for each WAVA solution.