Evolution of Expectations

Arian Evans, who leads WhiteHat's operations team, is in charge of all customer-facing aspects of our Sentinel Service. He’s tasked with ensuring our customer can track their current security posture no matter how many websites they have, how big they are, or how often the code changes. As such Arian has a front row seat to ongoing vulnerability assessment conducted on the worlds largest websites, week after week, month after month, etc. on a scale that no one else can touch. The other day he said after a lot of customer interaction that an interesting progression consistently take places. A set of phases where a customer’s business needs mature and expectations evolve when it comes to website vulnerability management. Arian drafted the outline and I filled in the fine print.



1. Quantity phase -- where more is more
The first phase of website vulnerability management typically begins by comparing solutions, often in terms of # of vulnerabilities discovered. Without the necessary expertise on staff to properly evaluate the results, unfortunately the solution displaying the most blinky-red-lights seems the most effective. And of course reports hundreds of pages in length must be more valuable than those only containing a few dozen (notice the sarcasm). That is until the time comes to actually parse the data and fix any of the issues whose existence is questionable.

2. Quality phase -- where less is more
During the second phase many figure out that certain website vulnerability assessment solutions, like scanners, come with a high false-positive/negative rate and a large duplicate problem. Reporting on the exact same XSS/SQLInj 10^5 times really doesn’t help an organization solve their problem. Then very soon afterwards it becomes clear that you can’t fix all vulnerabilities all at once. Eventually vulnerability remediation needs to be scheduled and prioritized in line with the general software release cycle. This is another reason why having a big list of vulnerabilities, even if they’re all real, without an accurate severity and threat ratings assigned is nearly useless.

3. Actionable phase -- how do I fix/improve things going forward with this data?
No matter how accurate the vulnerability assessment reports, fixing bug after bug after bug becomes tiring. The third phase of evolutions encompasses a strong desire to get the root of the problem. Customers begin asking why the same vulnerabilities keep popping up. Fortunately the data to address the problem may be already in hand, but only if you understand how to digest the reports. In our experience if a website has more than a few vulnerabilities of one class (XSS/SQLInj, etc.) and not many of the others, chances are the development framework is lacking. A few configuration changes or a version update could go a long way. On the other hand if vulnerabilities are spread out across several classes, mistakes are probably due to a lack of developer education or motivation. Having developers attend a class on secure coding best practices tends to have the most positive impact.

4. Consistency phase -- how do I do this consistently across time, because my software is always changing, without spending a zillion hours doing it?
Websites change, a lot! The responsibility for the security of more than say 3 websites (not to mention dozens or hundreds) easily becomes a full-time job, even if all the reports are perfectly accurate and prioritized. So the fourth phase has a lot to do with the importance of time management. Keeping up with website login credentials, scanner configuration, and simply pushing “go” every week is time consuming to say the least. Oh, and who is going to complete the remainder of the assessments that scanners miss? Dolling out fix-it tasks to various development groups and keeping track of remediation status should not be overlooked. At this point the most important thing to the customer is not about who finds more vulnerabilities (though its still important), but how dos the solution help to manage the entirety of the website vulnerability management process and keep the organization in sync.