Monday, August 27, 2007

The Big Picture

At SANS Ryan Barnett (Breach Security) and I talked a lot about the various suggested Website security “best practices” such as SDLC, black box testing, source code reviews, WAFs, scanners, etc - weighing their pros, cons, and costs. In the course of the conversation Ryan commented that I tend to look at the landscape from a scalability perspective. I hadn’t noticed this before, but he’s absolutely right. I try to look at strategies and solutions with an eye for the big picture with consideration for how many people/companies/websites the best-practices may benefit or not. If “best practice X” were rolled out on 1,000 or 10,000 or 100,000 websites…
  • What problem(s) does it solve and cause?
  • How much time and effort would be required for implementation?
  • Are there enough skilled people available?
  • What will it cost and who would be willing pay?
  • How much more secure will it make a website?
Etc.

These data points help develop insights into the where the industry is headed, the challenges we’ll face, and where innovation is most helpful. Plus I find it fun. To illustrate what I mean, below are some speculative high-level numbers I’ve collected which are used calculate “best practice” costs. Hopefully my math is accurate.


Important Websites: 60,000
According to Netcraft there are roughly 128 million websites. Of course not all of these are “important” (containing sensitive data, brand sensitive, etc). Netcraft also says there are over 600 thousand SSL websites, which is another useful metric, since why buy a certificate if it isn’t at least somewhat important. There could still be important websites not using SSL and neither accounts for intranet or access restricted websites. But because the number is still so large, I decided to stay conservation and take only 10% of the SSL total and use that moving forward. Figuring out how to properly secure the top 60,000 websites would really be something.


Vulnerability Assessments
The standard one-time black box vulnerability assessment (with or without the aid of tools) conducted on a single website performed by a qualified Web security pen-tester.

Required man-hours per assessment: 40 hours
Bill rate: $250 (per hour)
Cost per website: $10,000 (40 * $250)
Max number of assessments per year per person: 40
* Estimates are based on the data collected from the Web Application Security Professionals Survey and sanity checked through other sources.

To perform a vulnerability assessment on 60,000 websites each year requires:

Total man-hours: 2,400,000
Qualified pen-testers: 1,500 (websites / 40)
Total cost: $600,000,000 (websites * $10,000)
*These numbers do not take into consideration that many website change rapidly and may require multiple assessment per year.


Source Code Reviews
The standard one-time source code review (with or without the aid of tools) conducted on a single website and performed by a qualified software security expert.

Required man-hours per source code review: 80 hours
Bill rate: $250 (per hour)
Cost per website: $20,000 (80 * $250)
Max number of Source code reviews per year per person: 20
*Estimates are based on data contained within the techtarget article entitled, “Inside application assessments: Pen testing vs. code review”, and sanity checked through other sources.

To perform a source code review on all 60,000 websites each year requires:

Total man-hours: 4,800,000
Qualified source code reviewers: 3,000 (websites / 10)
Total cost: $1,200,000,000 (websites * $20,000)
*These numbers do not take into consideration that many website change rapidly and may require multiple source code reviews per year.



Web Application Developers: 300,000
It’s difficult finding a reference for the number of Web developers worldwide, so I figured I’d try JavaScript instead since most if not all of them know the language. According to a “JavaScript: The Definitive Guide advertisement”, there “300,000 JavaScript programmers” took advantage of the book. There are probably more Web developers out there who don’t know JavaScript, but let’s try to stay conservative. Imagine all the Web application code their churning out daily, let alone annually.

Secure Programming Training (2-day course)
* Based relative to SANS pricing information

Per Person: $2,000
Qualified trainers: 375
* Assuming 1 trainer is capable of conducting 20 classes per year with 40 students in each class.

To train all web application developers once per year:

Total cost: $600,000,000 (300,000 * $2,000)
*Does not account for any travel costs


From these numbers many takeaways can be derived, but here’s one that stood out to me:

Clearly more code is being churned out than our ability to assess it. Which means vulnerabilities will be pushed no matter what because the business is not going to wait around for security’s backlog. And if the bad guys just need to find one vulnerability, then we’re going to lose the battle. In fact the only things holding things together is there isn’t a critical mass of bad guys with the skill set to full exploit the opportunity. However, this will only last a short while and the smart money says attacks will continue to increase.

14 comments:

dre said...

Cost of a developer tester: same price you're paying him now. Cost of a developer reviewer: same cost you're paying him now. Cost of a moderator: give this guy/gal a 6% pay raise.

Cost of CT-Eclipse with PMD and EMMA: download time.

Cost and setup-time of Buildix: as long as it takes to burn and boot the CD.

Cost of JIRA (issue tracking) + FishEye (issue/repo searching) + Confluence (wiki) + Clover (C2 code coverage tool) + Crucible (continuous code review) + Bamboo (continuous build server) software products and integration to your existing environment (SVN, ANT, PMD, FindBugs, Canoo WebTest) when you want to replace the above free tools: Call Atlassian and hire a build/release engineer familiar with CI if you don't have one already. Outlook: very positive.

Outlook on WhiteHat, Cenzic, HP, and IBM's fault-injection offerings: price-less.

Give your job up as a code reviewer or vulnerability assessor / pen-tester because the market has a new solution that is a lot cheaper.

Jeremiah Grossman said...

dre, thank you, you just ruined my weekend. Now I gotta go research all that stuff. And I agree that giving the developers the tools and process to develop secure code makes a huge amount of sense. Two questions though.

1) Are you suggesting that if a company did all that you were saying, there is no need for security review by a third-party?

2) How would your advice serve those who do not have the set-up you describe and who already have dozens, hundreds, and sometimes thousands of vulnerable website already?

Anonymous said...

Hey Jeremiah, when you guys finish penetration testing, how do you guys generate a full report if you are using different resources, scanners, manually working?

Jeremiah Grossman said...

Hi Yousif, I really didn't get into describing our solution or what it costs. In this context it didn't really make sense to do so. But to answer your question ... our reports are generated automatically and displayed via web-base interface, something like Google Analytics if you will. All scanner found vulnerabilities are placed in a database, as are any vulnerabilities found by hand or otherwise discovered outside our technology framework. This way we can spend all our time finding vulns instead of drafting up reports and fighting with MS Word.

dre said...

1) Are you suggesting that if a company did all that you were saying, there is no need for security review by a third-party?

Yes, that's exactly what I'm saying. However, I also see a future where partners, extranet providers, SaaS vendors, merchants/CCP's, et al have to mature to certain [probably different] levels of independent, open evaluation security criteria. This is highly dependent on the nature of the web application. For example, if you provide a commercial, closed third-party web service, then there might be a secure software contract annex in place for each of your customers, especially if they are financials.

Of your 60k "important" websites, how many do you think would have this sort of working relationship? How many would want to handle it all by themselves - sans third-party?

In the meantime, PCI will force that requirement for L1 merchants and L1-2 service providers in the form of a yearly QSA code review... and for all merchants - a quarterly scan by an ASV.

Certain other compliance and working-relationship requirements can still bring in external review/testing as well.

Honestly, my suggested method is far more advanced than anything any PCI vendor (or other external review/testing company) is doing today. So requirement vs. nice-to-have is almost an arguable point, which is why I'm presenting it this way.

2) How would your advice serve those who do not have the set-up you describe and who already have dozens, hundreds, and sometimes thousands of vulnerable website already?

I firmly believe that my advice scales to these equations, in particular because of the costs and minimal required setup.

The most difficult part (besides implementing/forcing the process and culture) would be converting to revision control. There aren't a lot of technical challenges, although there may be developers who don't know how to write a unit test, don't know how to do peer review, and fail (or refuse) to grasp the simple concepts of continuous integration.

Production mistakes in publically-facing websites are usually a problem with pilot-error. Usually this involves going outside of process *and* preventing the tools from doing what they are supposed to do. So in some ways, it should almost be considered malicious. You are your worst enemy.

This also depends on what you mean by "vulnerable website". Vulnerable in what way and to what extent? My advice doesn't exclude incident response, which is probably what any internal security team or outside hired help should be spending their time/money/resources on. These people should be tracking down threats instead of finding new vulnerabilities. All they need to do for QA/dev is to provide them with coding standard requirements, possibly some cheat-sheet or taxonomy of errors documents (MITRE CAPEC and CWE should work fine) - and spend some time on formal/informal threat-modeling for new projects. I like the idea of empowering the people who are making the mistakes in the first place.

I'm mostly trying to change around the ideas of who tests/inspects what, when, and using which method/tool. Eventually it will go deeper than this, but I think I explained it well in the few short paragraphs in the FP.

Anonymous said...

@dre

Oh, you mean like current "mature" fields? Like, say, how architects' plans don't need to be reviewed by the building department, and consumer electronics don't need to be reviewed by the Underwriters' Laboratories, and contracts don't need to be reviewed by lawyers? Ah, I get it now, thanks.

Also, I didn't *quite* laugh out loud when I saw you mentioned the qaurterly PCI scans, but it was a near thing. Hacker Safe, indeed.

I guess I have been trolled.

Anonymous said...

Jeremiah, is there anything you can help me out with that will also adjust my company to the same process of generating a full report? I find small things and big things here and there and are pulled out of so many different resources, what is it that generates all this? Do you guys input this? Or? and if anything, can you help me adapt to the same environment of vulnerability generating? I too do not like to use MS Word or something associated because it's time-consuming and very unprofessional. help?

Jeremiah Grossman said...

Yousif, our backend reporting framework is proprietary to us, developed over the last several years, and part of our Sentinel service. No one outside of WH has access to it. Basically when the scanner finds a vuln, it saves the class of attack, the URL, input param, value injected, time of discovery, etc etc... then solutions, descriptions and examples, etc are provided automatically by the system. If the vuln is found manually, custom solutions and descriptions are written, but all the templateing is again provided by the system.

Most consulting shops in the industry utilize their canned text and MSWord templates, with a whole lot of copy/pasting involved. Still a lot of findings are custom written though. I've heard of several software reporting projects for security reporting try to get off the ground, but they've never gone anywhere.

My advice would be to try and get your hands on a few reports (MSWord or otherwise) from vendors .... note the things you like and dislike... then improve upon their work. Always the central key is focusing on addressing customer needs.

Anonymous said...

Jeremiah,

A few of your recent posts have talked about the issue of a "critical mass" of skilled crackers with a criminal motive. This important concept pops up frequently when people write about information security, usually accompanied by a prediction that the balance is shifting in the direction of the forces of evil. I don't disagree. Larry Rogers in 2000 wrote a paper for CERT titled "Means, Motive, and Opportunity" where he talks about the same thing.

You ask a VERY interesting question. Given that we industry insiders know how easily cracked is the average site, why are we NOT seeing a huge increase in the number of significant, successful attacks? I've thought about this quite a bit over the years and my reasoning takes a different line, one that many in our industry may disagree with.

I believe that we have no shortage of people having both the Means and the Motive. I think what's lacking is the Opportunity. Despite impressions to the contrary, the Internet is not a "target-rich environment" (to use Pentagon-speak) for important data.

Yes, there is a market for stolen credit card information. But stolen credit card information has been commoditized to the point where it tends to attract only small-time criminals.

It seems impossible that the Internet would not be a target-rich environment for crackers, right? Actually, I believe that organizations over the last 7 years have put much LESS of their important data on the Internet than was originally predicted. The prevailing prediction in the late 90's was that every company would be using Internet content management systems to implement portals and B2B applications to exchange business data with customers, vendors, and partners via standardized exchange formats based on XML.

Well, this hasn't really happened to anywhere near the extent that we predicted back in the 90's. I think there has actually been a retreat from deploying Internet applications where significant business data is involved. This is due in large part due to the industry's recognition of the real security risks involved.

Now I'll make a dire prediction of my own. :) The pendulum is swinging again, with the Software as a Service (SaaS) revolution. How many companies are storing their crown jewels in the likes of Salesforce.com and NetSuite? The number of competitors is proliferating. New service models are being built upon new architectures and people are signing up en masse.

I think SaaS signals the beginning of a target-rich Internet, which provides the "opportunity" in the triad of Means, Motive, and Opportunity.

Anonymous said...

I know no one can have access to it, but I like the idea of it and how it works, do you mean if I used a scanner like Acunetix and finished, besides Acunetix's report, does the sentinel thing automatically take what it needs to OUT of the report? and if the program doesn't generate a report, does this sentinel thing just pull it out SOMEWHERE in the program? if so, I mean is there ANYTHING else you can help me with? MS WORD is not so professional.

Anonymous said...

HI Jeremiah, you say

our reports are generated automatically and displayed via web-base interface, something like Google Analytics if you will. All scanner found vulnerabilities are placed in a database, as are any vulnerabilities found by hand or otherwise discovered outside our technology framework. This way we can spend all our time finding vulns instead of drafting up reports and fighting with MS Word

Most of the tools generate reports in English. Do you present your results to your customers in English? It might be a problem if your customers insist on other languge.

Jeremiah Grossman said...

Hi Chad,

Wow, I had not even considered that line thought of thought. I have a hard time wrapping my mind around that concept though. There are a great deal of targets, but yes, does that mean its target rich? Or it could also mean that its not perceived as being target rich from bad guy point of view.

Might we consider that the environment is target rich, but the bad guys don't yet know how to identify the targets or monetize on the information readily available. I mean, there is no shortage of stock moving information that one could obtain in secret. Same with SaaS like Salesforce. Or juice data contained in webmail. The list goes on. Could it be they lack creativity and simply don't take how to take advantage of the situation.

Whatever the case made be you definitely gave me something to think about. And yes, the SaaS movement is coming... every business is going to be reliant upon another's security.

Jeremiah Grossman said...

Yousif,

Its real simple actually, our customers log-in to our Sentinel portal and they see their current vulns in real-tme. Should customers desire to export the data they may do so in PDF, HTML, and XML.

This is not too different to what a desktop scanner like Acunetix, AppScan, or WebInspect do for their reporting. About 50% of consultants use scanner generated data to supplement their reports. Scanner->Word->PDF.

Word documents can be made to look very professional if you put some work into your templates. My advice for templating or auto-report generation software for webappsec is to ask the people on the mailing lists. I'm not the best person for this since Im somewhat removed from the consulting realm.

I hope this helps.

Jeremiah Grossman said...

juk, no not yet. English is the only option we currently offer. This has been OK for us so far since all of our customers either reside in the U.S. or speak english. When WhiteHat decides to look for opportunities in other countries, we've designed our systems in such a way thats makes localization easy.