If You Want to Improve Something, Measure It

 

During the past five years WhiteHat Sentinel has performed comprehensive vulnerability assessments on thousands of websites. These assessments are ongoing, as performing continuous assessments is integral to our core values at WhiteHat. With each assessment we analyze the results − discovering new facts about Web application security, share that knowledge through reports, and receive feedback from companies about which information has done the best job helping them secure their websites.

Now that we’ve gathered this large amount of data on website security, we believe that any security professional can use the seven metrics − all measurable and mostly automatable − listed below to track the performance of their application security program and the software development lifecycle that drives it.

The Seven Key Application Security Metrics

1. Discoverability – The Attacker Profile
What are the risks of a vulnerability being discovered, and what type of attack − Opportunistic, Directed Opportunistic, or Fully Targeted – will the attacker use? Does the attacker need to be authenticated in order to breach security? Is the damage that the attacker can cause within the risk tolerance of the organization attacked?

2. Exploitability – The Difficulty of Attacking
Are the site’s vulnerabilities possible to exploit? Theoretically? Possibly? Probably? Easily? Are proof-of-concept exploits available for demonstration purposes? Is it becoming more difficult for attackers to exploit issues previously identified? That is, from day to day are there less trivial vulnerabilities per application, and are the remaining vulnerabilities less likely to be exploited?

3. Technical and Business Impact – The Severity of the Attack
What negative impacts can the vulnerabilities have on the business – both technically and financially? If a vulnerability is exploited, is the breach communicated within the organization and is an estimate made of the possible damage – quickly? Once an attack is discovered will business stakeholders and development groups within the company prioritize the risk and take action to remediate the problem accordingly?

4. Vulnerabilities-per-Input – The Attack Surface
How common and how deep across the codebase are the vulnerabilities exploitable? Does the codebase become more secure – or less secure – as new code and acquired code are added; as legacy code requirements increase; and as new technologies are introduced?

5. Window of Exposure – The Exposure to Attacks
How often is the organization exposed to exploitable vulnerabilities: Always, Frequently, Regularly, Occasionally, or Rarely? How effective are the quarterly or annual remediation efforts? What is the best method for shortening the window of exposure time, i.e., the risk of compromise? Is the risk diminished by reducing the number of vulnerabilities being introduced? By increasing remediation speeds? By fixing more of the system security issues?

6. Remediation Costs – The Opportunity for Greater Savings
What are the costs of fixing the vulnerabilities that are discovered? Over time, is the cost-per-defect to remediate the vulnerabilities decreasing? Is it possible to decrease “lost opportunity costs,” i.e., to reduce the time that developers spend “fixing” security issues rather working on new business projects? And how can the efficiency of developers’ remediation efforts be improved?

7. Defeating Vulnerabilities – The Focus on Identifying Their Source
What type of code or which business unit is introducing the vulnerabilities? Are the vulnerabilities internally developed, third-party developed, from acquired software, etc.? When multiple vulnerabilities are discovered, where should the focus of attention be on immediately? Can the level of software security be improved through developer training, contractual obligations, software acceptance policies, etc.?

“To Improve the Process, Measure the Process”

In summary, corporate security teams often tell us that their businesses need to become much more concerned about their Web application security. I’m sure many of you readers can relate to such comments. At WhiteHat we’ve learned that the three primary obstacles to establishing a successful website security program are: (1) compliance issues, (2) company policies, and (3) a lack of data that will support the need for greater security.

While compliance issues and company policies may sometimes work to get management’s attention, I believe that data – hard numbers – work much more effectively.

When management understands how secure – or unsecure – its websites are, how secure they have been in the past, and how the company’s current Web security compares to the security of its competitors, then almost certainly the resistance to devoting more resources to Web security will diminish. Furthermore, there will be new respect within management for both the Web security services provided and the teams providing them.

As the old saying goes, “If you want to improve something, measure it.” Because almost anything that can be measured will improve. Which of the 7 metrics discussed here are you measuring today? Which of these metrics are you currently unable to measure at all? And are any of the important metrics presented here missing from your list?  Or maybe you have new ones to add!