Thursday, December 11, 2008

Budgeting for Web Application Security

“Budgeting” is a word I’ve been hearing a lot of questions about recently, which is another data point demonstrating that Web application security and software security are increasingly becoming a top of mind issue. The challenge that many security professionals face is justifying the line item expense for upper management. Upper management often asks, “How much do we need to spend?” well before “What do we need to spend it on?” I was talking with Boaz Gelbord (Executive Director of Information Security of Wireless Generation) and several others about this, and they provided keen insight on the subject. I have identified the following approaches to justifying security spending:

1) Risk Mitigation
"If we spend $X on Y, we’ll reduce of risk of loss of $A by B%."

2) Due Diligence
"We must spend $X on Y because it’s an industry best-practice."

3) Incident Response

"We must spend $X on Y so that Z never happens again."

4) Regulatory Compliance

"We must spend $X on Y because PCI-DSS says so."

5) Competitive Advantage
"We must spend $X on Y to make the customer happy."

Risk Mitigation
Risk mitigation is the efficient, forward thinking, and scientific approach to security. Many extremely bright people are pushing this methodology forward, especially through I’ve had the pleasure of rubbing shoulders with many of them at Metricon events including Andrew Jaquith, author of Security Metrics: Replacing Fear, Uncertainty, and Doubt, who does a stellar job of walking the reader through this world. The fundamental idea is to find an acceptable balance between the amount of investment and the risk reduction that represents "good enough."

While measuring investment against risk makes sense, it is also extremely difficult to quantify in Web application and software security given our industry’s lack of data. We don’t yet have strong data tying loss to root cause and those compromised closely guard the details. Only recently, through WhiteHat Security and WASC Statistics projects are we getting a wide look at who is exposed to what. Also remember that just because a vulnerability exists, doesn’t mean it’ll get exploited. Possible != Probable. My hope is that this approach continues to mature and becomes a widely accepted reality.

Due Diligence
Due diligence is another approach to security budgeting that considers the ramifications of a data compromise if it occurs. Among other things, upper management will want to assure customers and business partners that they were investing resources into the security program in line with industry standards. At the same time, a budget executive may not approve a larger than average security budget in the event no security incident occurs during the fiscal year. This common scenario prompts the question as Boaz puts it, “What is industry standard for security spending in Web application security or software development?” For example, with regard to IT backend systems, there are studies that suggest 3-10% or higher for security spending out of the total budget. In this context, the "Security needs to be embedded into every part of software development" mantra isn’t helpful.

To the best of my knowledge there has been no formal study or survey on how much organizations are spending or should be spending regarding security in software development (or webappsec specifically). What most organizations typically do is outline the best-practices (architectural reviews, security QA, pen-testing, WAF, etc.) they should be implementing relative to peers. Next, they identify the appropriate solutions and obtain rates from the vendor.

Incident Response
Few things free up security dollars faster than a data compromise that is publicly attributed to the organization. That last part is really important. It is possible for an organization to ignore a general Web server compromise, which they may not even know about. For example, when the bad guys are only interested in using a company's resources to improve the Google Rank of some website (i.e. Al Gore’s Convenient Truth blog). Things get much different though when a revenue-generating website is SQL Injected and begins delivering malware to its visitors which has already affected millions of users. The result is the website gets blacklisted by the search engines and traffic sinks like a rock. Now we are talking real quantifiable losses that get people's attention.

At this point an organization is in triage mode. How do they best remove the malware, get off the blacklists (may take days or weeks), and continue business operations? This is also a good time for a security professional when they are given license to pursue projects that ensure a similar event never happens again. They’ll be given the necessary resources to be proactive. They then have the power to prioritize activities that may fall into the category of Risk Mitigation without having to justify “why.”

Regulatory Compliance
Regulatory compliance is a huge justification lever for security professionals. Compliance pressures enables them to do the things they want/need to and gives them the ammunition to justify their requests. For many security professionals, this is of course a very welcome development. And again as Boaz lays it out, “it is much easier for me to justify costs that are legally mandated instead of pointing to threats that in practice are not very likely to materialize. As a citizen/user it is also a positive development, since without legal requirements I think that the already poor state of software/application security would get even worse.” Still the challenge is that most regulations do not indicate how far organizations must take these measures so the mileage will certainly vary.

Competitive Advantage
Competitive advantages. Every organization needs them, and wants more of them. Security that can be demonstrated or made visible can be used as a competitive advantage. Sure, security adds cost to the goods, but that fact could be capitalized on if the customer assumes a level of risk. As Bruce Schneier says, “you can feel secure even though you're not, and you can be secure even though you don't feel it.” For example, the McAfee Hacker Safe or secured via SSL logos can be seen as one of those things that makes consumers “feel” safer when conducting e-commerce. A firm can also engage with a reputable security firm to get a security assessment that can be shared. Deciding which to go with is directly proportional to the customers level of understanding.

At the end of the day no one can say if any one way is the right way, and that was not my intention, but instead to document the approaches I’ve seen gain a better understanding of what works best for a given environment. In order for Web application security and software security to improve, more dollars need to be allocated, so we must assist the stakeholders in that process as best we can.


Richard Bejtlich said...

Great post!

Unknown said...

I think it is interesting and ironic that the founder of whitehat security has a blog that looks nearly identical to the blackfistsecurity blog. Anyway, great post. I have never been here before, but Bejtlich was pimping this on his blog so I stopped to check it out. I just had to comment on the appearance

Jeremiah Grossman said...

Many thanks Richard!

@Black First, Welcome, but not sure I get the irony. I guess we both enjoy the same Google template.

Anonymous said...

I think you really hit the nail on the head with the 5 reasons for security spending today. Security conscious companies produce secure code because it is the right thing to do and they want to be in pole position when regulations with teeth finally come around. Unfortunately, in most companies these 5 reasons alone do not end up justifying sufficient security spending:

(1) Risk mitigation – the estimates you hear of $200 per compromised record seem like an exaggeration in most industries. IMHO it is the absence of an economic motivation to produce secure code that is starting to lead to regulation.

(2) Due diligence and industry best practice - When I talk to software vendors and start asking too many specific security questions, I invariably get a list of their Fortune 100 customers, as if that in itself means the software is secure. The apparent effectiveness of the “our product is secure because big and important companies use it” argument has led to the status quo of insecure software.

(3) Incident Response - This one is definitely a motivator, especially when it involves visible compromise such as DoS. But only a small portion of software security incidents lead to measurable damage to the affected company, and most companies do not suffer from damaging incidents that can be attributed back to them.

(4) Regulatory Compliance- with the possible exception of PCI and a few industry specific regulations, current regulations are too vague and open to interpretation. Companies can still claim compliance while only paying lip service to security. This will change over the next few years (eg. new state law in Massachusetts).

(5) Competitive Advantage

This is the area where we as security professionals have the most work to do in helping consumers differentiate secure products from insecure ones. Security carries a cost, so that an insecure product costs less to produce than a secure one. Bruce Schneier often calls the software market “a market for lemons”- in the used car market, it is difficult to tell a lemon from a good car. It is very hard to differentiate secure software form insecure software, reducing the motivation to invest in security.

When addressing security, corporate marketing materials will often mention things like the biometric readers at the doors to the data center while saying nothing about secure coding. When is the last time a company suffered a public data breach by someone walking out with a server from a server room because the door was merely locked and did not have biometric passes? You also see SSL mentioned almost all the time. Quite a random example given the vast array of web application vulnerabilities.

When I am evaluating a vendor, I would much rather see “we spent X% of our budget last year on internal and external security reviews”. I have no way of knowing if this money was well spent or if the reviewers were competent, but at least there is an objective measure of effort and attention put into security. Having this additional metric will help make sure that the market accepts a premium for wisely spent security dollars.

Sorry for the long comment and thanks for bringing much needed attention to this issue through your (always enjoyable) blog. I am putting together a group of CISOs and security professionals to get some data on this issue. Participation is welcome and I will post a URL once the initiative gets rolling.

Anonymous said...

J- As always, good stuff. Let me ask you:

"While measuring investment against risk makes sense, it is also extremely difficult to quantify in Web application and software security given our industry’s lack of data. We don’t yet have strong data tying loss to root cause and those compromised closely guard the details."

I hear this often. What I don't hear is someone specifically saying - here's the amount and quality of information we would need, and a plan to get it.

Alternately, I'm willing to bet that in most cases there is enough information to make consistent, defensible decisions. It's the lack of models and understanding around probability theory that hinders us.

Jeremiah Grossman said...

@Alex: To your point, I believe we need insight into fraud rates, not just data disclosure/compromise rates. We hear a lot about X number of records being lost because a backup take went missing. What we don't know is if that loss led to fraud - that type or metric would be invaluable. We'd be able to take/advocate more exact security measures that truly reduce fraud and not just data loss.

Jeremiah Grossman said...

@Boaz, excellent comment, I have nothing to add. :) Please let me know you are ready to formalize that group of CISOs.

lennykaufman said...

This is a great post - I think it can be better ...

The one mistake that we constantly make in Information Security Risk Assessment is that we tie measurable factors (primarily $$) to immeasurable factors (unbound % or # wrt risk).

Your post is a good leap forward in laying out the domains of consideration, and it would benefit from taking the final step that Financial Risk Management methods take (which is why Financial Risk Management is so much more effective than InfoSec Risk Management, to be honest).

To use your example, a more complete quantification would be as follows:

1) Risk Mitigation
"If we spend $X on Y, we’ll reduce our risk of loss of $A by B%, resulting in $C financial upside for our organization."
(I've essentially only completed the formula that you set out.)

2) Due Diligence
"We must spend $X on Y because it’s an industry best-practice that has been shown to carry a financial upside of $C (in either cost avoidance or revenue generation)."

3) Incident Response
"We must spend $X on Y so that Z never happens again, saving us $C (which equals the estimated cost of incident response and/or incident-related loss."

4) Regulatory Compliance
"We must spend $X on Y because non-compliance with Regulation A carries an estimated cost of $C."

5) Competitive Advantage
"We must spend $X on Y to make the customer happy, because making the customer happy has an estimated financial upside of $C for our organization."

Jeremiah Grossman said...

@lenny, whoa! This is pure gold... excellent stuff. I'm thinking this post is deserving of a v2 in a couple weeks of assimilating community feedback. I'd like to add more linked resources, case studied, metrics, and of course incorporate your additions (if you don't mind).

lennykaufman said...

Don't mind at all. I'll watch for the links, etc. (and probably chime in with more feedback as things progress).

Thanks for starting this thread - awesome.

Anonymous said...


I'm confused (it's not difficult to do, confuse me). Are you saying that organizations don't have insight into the amount of significant fraud they have experienced?

Jeremiah Grossman said...

@Alex, oh oh I see what you are asking now. Yes, a company can and should know how much fraud they are experiencing. Though this information is not always readily available, but if it is, it certainly can be used to justify security spending. My comments were geared towards the industry at large, where we don't have good fraud rates.

Tom Brennan said...

Appplication Security #1 on the agenda.. is 2009 the the year of Application Security?,com_csis_pubs/task,view/id,5157

Unknown said...


Cool. I just hear that quite a bit, and I'm not convinced that fraud rates are relative priors unless we specify to the point of small sample size.

I.E. If I'm regional bank, the fraud rates of community or large national banks may or may not be relevant. My own history may be a better indicator of expected loss magnitude or the expected frequency of fraud events. Both pieces of information are useful, the question is how useful, which is a different conversation than the quantity of information at hand.

H. Carvey said...

As an active responder, what I'm seeing in the corporate community is that risk is viewed differently...spending now on security is a certainty, whereas the risk is a possibility. Many managers tend to bet on the horse NOT coming in.

"Due diligence" is one of my favorites. I get calls from folks who want me to come in, and acquire and analyze their entire network, just for "due diligence". It occurs to me that if you really wanted to do "due diligence", you'd've had someone in before-hand.

Regulatory compliance is HUGE! Some compliance measures have teeth, others only have teeth when it's the little guy.

Finally, competitive advantage...for the folks who do read the papers, there are enough stories out there about loosing intellectual property and then market share due to hacking, a disgruntled employee, etc. But then, it keeps happening, and will continue to happen, because managers have a "yeah, but that kind of thing won't happen to me" attitude.

Final word...the issue of "budgeting" is profoundly interesting to me, as I've responded to a number of engagements which would've been prevented had management required the admins to add or change passwords (SQL 'sa' accts, etc.). All this means is, sit down at a keyword and type a command. Yet this was deemed too expensive...the cost did not justify the action.

What business school did these guys (and gals) go to?

Jeremiah Grossman said...

@Keydet89 As you said, everyone like to see security justified in a different way, even when its a contradiction in terms. This is probably going to be the reality of the situation for the foreseeable.

Business school. Wish I had the opportunity, would have made life a bit easier. I had to learn some important lessons in the field at great expense.

FJ said...

I agree with the "top 5" but are any of these really doable unless you can actually see and control all these new web apps? I mean, port 80 is a black hole and 443 is encrypted. And IPS isnt't the answer for seeing all this stuff. Any thoughts?

Jeremiah Grossman said...

Franklyn, money first, control second. :)

Gary said...

Hi. I don't understand why you are pimping up application security specifically: the 5 methods you describe apply to a whole range of security/risk management investments.

I can think of at least two more to add to your 5:

6. If we didn't spend $X on security, we wouldn't be able to perform business processes Y and Z safely and/or profitably [i.e. security as a business enabler]

7. If we spend $X on security, our governance, compliance and risk management objectives will be met, our breach costs will be contained, but most of all our executives will sleep soundly [i.e. the assurance factor].

Kind regards,
Gary Hinson

Jeremiah Grossman said...

@NoticeBored, because Web application security is what I spend most of my time on. But you are probably right and the guidance could apply to many other areas.

#6 is pretty close to #1. At least I can't discern a big difference.

#7, seems to be a mix of #1 and #4

lennykaufman said...

You mentioned a v2 ... any word on if that's a go?

Jeremiah Grossman said...

Been working on the OWASP Security Spending Benchmarks project with Boaz. Out of the data collected from that will probably fuel v2.

lennykaufman said...

Coolio. Let me know how I can get involved with the OWASP Security Spending Benchmarks project.

Jeremiah Grossman said...

@lenny, please pitch me an email directly. Much of the work has been done, but I have a feeling there is going to be a lot more post data collection.

Anonymous said...

what a great post. I really enjoy reading this post.

Juan Carlos Calderon said...

It might be part of competitive advantage but IMHO a #6 (and maybe higher in the list) is actual Money savings (Hard and soft). When you implement a security program adding tools in early stages, starting to use common and approved components and that kind of activities, that has an immediate savings effect. You have less bugs to fix before deployment, you fix design errors at design (one of these might get you yo a total rebuild of the application), you have better TTM, developers learn early on common vulnerabilities so they fix 1 not 100 XSS. The though part is how to measure this, although a simple baseline comparison by doing some testing in a few applications and your target objective might be helpful to figure out the numbers.