Recently Alan Shimel (StillSecure) went out on a tiny twig and said, “vulnerability assessment (VA) is dead”. Of course Alan’s speaking about network security not web applications. His remarks are about VA's convergence with NAC’s. Fair enough. When I spoke with him he said, “Actually VA for web apps is one of the few bright spots in the VA space these days.” I'd like to think so. :) This topic is always on my mind since this is exactly what my company does. “What is the future of web application vulnerability assessment?” is a question that doesn’t get asked a lot. Personally I think we’re at the point where network VA was a few years ago, solving the challenge of scaling.Where are we now?
- 105 million sites are on the Web with 4 million new ones each month.
- Perhaps hundreds (?) of thousands of websites collect or distribute personal information, financial and healthcare data, credit card numbers, intellectual property, trade secrets, etc.
- Web application issues top every major Top-X vulnerability list.
- 8 out 10 websites are full of holes and most of the attacks are targeting the web application layer.
- Assessments should be performed after each code change or "major" release and require about a week or two of human-time to complete.
Analyzing the scope using some assumptions:
- 500,000 “important” websites (roughly 1/2 of 1% of the total population)
- Assessments 2-times a year per website. (Vary on change rate)
- An expert can perform 40 assessments per year with base salary of $100,000 (US).
- Retail cost per assessment $5,000 (US). (Normally higher ranging between $8,000 - $15,000)
Today we'd need:
- 1 million total vulnerability assessments
- 25,000 experienced experts in web application VA
- $2,500,000,000 (US) in salary for web application experts
- $5,000,000,000 (US) retail assessment cost
Of course as the awareness of web application security builds the numbers will climb, but for now we have to face facts. And the fact is unless we can vastly improve the web application VA process, most websites will not be assessed for security and remain insecure. That’s what’s going on today. And that’s why I’m saying the future of web application vulnerability assessment is about scale.
While we certainly can’t reduce the number of “important” websites, can reduce the number of man-hours and expertise required to perform an assessment using technology and a modern processes. Modern assessment processes need to be highly streamlined, repeatable, thousands running concurrently and performable by less than top-tier webappsec experts. This is what it truly means to “scale”.
How much improve can be made near term is a subject of much debate, but we’re working on it. For fun, let’s try a few more guesses at how certain efficiencies will help.
Future improvements:
- 500,000 “important” websites (roughly 1/2 of 1% of the total population)
- Assessments 2-times a year per website. (Vary on change rate)
- An expert can perform
40200 assessments per year with base salary of$100,000$80,000 (US). - Retail cost per assessment
$5,000$2,000 (US).
- 1 million total vulnerability assessments
- 5,000 experienced experts in web application VA
- $2,000,000,000 (US) in salary for web application experts
- $400,000,000 (US) retail assessment cost
8 comments:
And to top it all off, the most serious of the vulnerabilities are impossible for any automated tool to find.
Source code scanning earlier in the development process and more reliable VA scanners will help, but the lions share of my VA time is spent working on logic flaws that neither a source code scan nor an automated application scan can find - horizontal and vertical privilege escalation (some will tell you their tools can, but they're not that great), XSRF, and privacy issues.
What will help to a degree is for coders to be more focused on coding correctly (no, we don't have to train them to use MITM proxies against their sites in order to do this) and for engineers to understand that security needs to be baked in from the beginning.
We're probably in a losing proposition, though. The bad guys always have an unlimited supply of able workers and the reward scale is such that money isn't necessary (initially) to motivate them.
> And to top it all off, the most serious of the vulnerabilities are impossible for any automated tool to find.
A lot of em, that's for sure. I wonder how many people are getting this part of web app sec yet.
> What will help to a degree is for coders to be more focused on coding correctly
While improvement will inevitably make websites more secure, it doesn't reduce the workload for web app VA. We have to check for the same things anyway, by scanner and human.
> (no, we don't have to train them to use MITM proxies against their sites in order to do this) and for engineers to understand that security needs to be baked in from the beginning.
Well said.
>We're probably in a losing proposition, though. The bad guys always have an unlimited supply of able workers and the reward scale is such that money isn't necessary (initially) to motivate them.
Taking the Web as a whole, your probably right. Far too many websites to protect, far too many vulns at the moment. Its a cake walk to break into just about anything you want.
Where we need to get to is being able to offer those that REALLY want to be secure the means and solutions to do so. The rest who don't care, well, not much can be done.
I don't understand your math. You approximately halved the retail cost of an assessment, but the price went down an order of magnitude. I think the $2,000/assessment retail is incorrect, given that worker productivity went up over 50-fold. Did you mean to type in $200?
From nits to useful comments, I think that a lot of the reason that network assessments are cheap now has to do with the very things that make people say that web assessments are hard. Networks tend to be standardized, and net admins are educated about security. The parallels to this in the web app world are frameworks and education. Un/fortunately, I don't see either of these being at the maturity level of networks anytime soon. Network architecture has essentially stagnated. Nobody's deploying any radical new network architectures. Wireless would be the exception, and we can see how secure that is. I don't think web assessments will be as straightforward as net assessments until web development stagnates and frameworks emerge that are as widespread and secure as TCP or SSL implementations are today. With all the churn and parallel implementation that exists in web development today, I don't see that happening anytime soon.
(P.S. I get like a 50% success rate tops on your captcha. Does that mean I'm a replicant?)
> I don't understand your math. You approximately halved the retail cost of an assessment, but the price went down an order of magnitude. I think the $2,000/assessment retail is incorrect, given that worker productivity went up over 50-fold. Did you mean to type in $200?
In my model worker productivity increase 5-fold (from 40 assessments to 200) which could have meant an retail price decrease from $5,000 to $1,000. However, I don't think the two metrics are necessarily directly related. $1,000 was too cheap. I just picked some numbers that felt right based upon the trajectory I see at WH.
> Networks tend to be standardized, and net admins are educated about security. The parallels to this in the web app world are frameworks and education. Un/fortunately, I don't see either of these being at the maturity level of networks anytime soon.
That's a very compelling observation. That could very well be exactly what's going on. And frankly, a topic article worthy if expounded upon. I'll have to think about this more.
> (P.S. I get like a 50% success rate tops on your captcha. Does that mean I'm a replicant?)
HEHEH, I've had the same problem and wonder the same thing often about myself.
Oh, wait, I see the math problem. You swapped the assessor cost and the retail assessment cost in the "brave new world" scenario. It still costs billions of dollars to do the tests, but at least you can do it with only 5,000 people.
Yah, just trying to put things into perspective, even though numbers will be somewhat innaccurate.
Hi Jeremiah, I'm a security professional for a major and am digging around the web for basic metrics around patch management and the cost associated.
I've seen some complex math formulas that don't benefit what I need.
I was looking at the cost of IT operations for testing, deploying a patch on 100,000 endpoints with 20% dev/test machines. Heterogenous environment mostly Windows, with 10% *Nix, 10% network devices.
Based on premise that enterprise deployment tool is in place for Windows and that the testing would involve manual patching for the 20% of the dev machines.
@Anonymous: That's a really good question with important data. Unfortunately, I don't have a good reference to the data that you need, but suspect that someone out there does.
Are you on twitter? If so, that would be a better place to ask. I'll RT if you point it out to me.
Post a Comment