Recently on Twitter I asked why some people feel oddly compelled to rely upon the shortcomings of Web Application Firewalls (WAFs) as a means to advocate for a Secure Development Lifecycle (SDL). To me this is odd because the long-term, risk-reducing value provided by secure code is enough on its own to warrant the investment. If you can’t demonstrate that, blame directed at WAFs seems misplaced. Most importantly, we must remember our objective: protecting websites from being hacked.
The ultimate scope of this objective, 200 million websites and counting, cannot be understated. Even if we just focus on the 1.3 million websites serving up SSL certificates, the scale is still unbelievably massive. Whatever the metric, experienced industry experts and aggregated statistics reports agree, the vast majority of these websites are riddled with vulnerabilities. The exploitation of thousands of websites that is fueling headlines serves as a further proof point. To quantify vulnerabilities, let’s assume an average of six serious* vulnerabilities per website, WhiteHat Security’s published figures based on our own Sentinel assessments. This totals 7.8 million custom Web application vulnerabilities in circulation. We just don’t know exactly where they are.
The next and equally important problem, fixing the code, is a seemingly insurmountable obstacle. Imagine an extremely limited number of application security pros to convince 17 million developers (some unknown portion being Web developers) to add to their workload, learn about defensive programming, and remediate all the vulnerable code. And by the way, this will be accomplished in small increments. Vendor-supplied patches have no place here. According to Gary McGraw’s (Cigital, CTO) BSIMM studies, observations from large-scale software security initiatives, a software security group (SSG) ideally should be 1% of the size of the development team. Given that baseline, we’d need 170,000 software security experts when I doubt if more than 5,000 currently exist.
To pile on, process and staffing investments will not take place unless the stakeholders are convinced that it is worth it to risk potential revenue by refocusing developer efforts to fix security issues that may or may not be exploited (aka: risk management). This is a scenario application security practitioners are all too familiar with and frustrated by. Despite all the challenges, software security is making significant progress compared to its nearly nonexistent status of only a few years ago. However, change does not occur over night. It takes years. Unfortunately time is not on our side and the bad guys are exploiting websites and their users by the thousands every day. Waiting around for a future of software security utopia while the losses mount is ill-advised. Obviously we need more options. Just yelling for more manual code reviews, developers training, secure frameworks, etc will work for some, but certainly not all. Think about the costs!
Several years ago I was contemplating the sheer size and gravity of the aforementioned numbers, taking into account operational considerations, brainstorming possible solutions worth considering, and then it hit me. I disliked WAFs for all the same reasons as everyone else, but it became clear that even if WAFs didn’t work, practically speaking we really need them to (work). If somehow they could temporarily protect against the exploitation of a certain amount and type of vulnerability, and do so without requiring an application security person to beg for development time, allowing them to take action alone, it would have tremendous appeal. Don’t agree? Remember, WAF technology predates PCI-DSS and hundreds of millions in annual sales is not entirely driven by compliance.
Deciding to be part of a solution, or at least an option for organizations to consider, I began meeting with various WAF vendors asking how WhiteHat could help. One idea we considered was that if a WAF knew ahead of time the exact type and URL location of a website vulnerability, then perhaps WAF effectiveness would improve. Thus our exploration into virtual patching began. Let’s be clear though, WhiteHat Security does not make WAFs. We do not sell WAFs. We have no financial motivation when or if they are sold. Our main interest is assisting our customers in protecting their websites and integrating what we do into technologies they feel are beneficial, to them.
Fast forward to today. Obviously not everyone is impressed with the current state of WAF technology. Staunch critics say WAFs introduce the very vulnerabilities they are supposed to defend against and/or that they can be bypassed. We agree. No doubt, WAF technology is imperfect, but that doesn’t make them any different from every other security product segment. Anti-malware, firewalls, IDS/IPS, MFA, SSL, and so on all have vulnerabilities. All can be bypassed. Still, I’m pretty sure we’d all agree that these products provide a certain amount of value nonetheless. You don’t throw the baby out with the bath water. We regularly identify vulnerabilities and bypasses in various WAFs and results with the vendors so they may have an opportunity to improve.
So where does the anti-waf-software-security-only-zealotry really come from? I don’t believe the critics are ignorant of the problem at hand. Maybe it has something to do with the quote from the movie Vanilla Sky, “What’s the answer to 99 out of 100 questions? Money.” Technically speaking, SDL activities and WAFs are NOT mutually exclusive. They are designed to solve different problems. It is entirely reasonable to deploy one, the other, or both at the same time. The tough reality is that IT security budgets allocated towards Web security are wholly inadequate. Budget constraints force organizations to make painful choices about where to spend first. Decisions are sometimes made for compliance reasons, and other times for security. While SDL and WAFs should not be considered competing solutions, they do compete for the limited dollars. PCI-DSS 6.6 made matters that much worse.
No matter the reason, when WAFs are purchased “first,” the software security guys lose money, naturally making them resentful. They snap back saying, “the customer doesn’t get it” and “SDL is the right thing to do because WAFs suck!” Let’s be honest, if that’s how you are positioning the value of an SDL, you deserve to get back-burnered and furthermore you’ve done your client a disservice. WAF’s suckage should have zero bearing on whether or not an organization should improve its SDL. Instead, focus on the many cost-saving, risk-reducing, top-line-benefiting qualities an organization may realize by implementing a well-regulated software security program.
At the end of the day, our common enemy is really the lack of application security visibility and the allocation of necessary resources. If we come together and help address this as an industry, we’ll all be better off, and the pressure of this either or choice will be lessened.
15 comments:
I think some of the antiwaf sentiment came from the overzealous marketing from the early companies in the field. There are still a number of vendors claiming that a waf can completely solve the OWASP Top 10 which is ridiculous.
That said, far too many in our field simply can't handle things that aren't perfect protection. This kind of black-and-white thinking harms our credibility with business immensely.
There's one waf use case that seals the deal for me. The capability to quickly deploy a 'virtual patch' is something that every organization should have in place. We built the OWASP ESAPI WAF so that anyone could easily deploy this protection right in their application.
@jwilliams, that indeed could be the case, carry over from the early days.
You know, in the past I publicly called out scanner vendors for their loose OWASP Top Ten marketing claims on a number of occasions. I'd have no problem doing the same for WAF vendors who cross the line.
If anyone has URL references to such behavior, send them my way.
FYI, this paper shows WAFs are subject to OWASP top 10 attacks themselves:
http://www.h4ck1nb3rg.at/wafs/final_project_documentation.pdf
I think the WAF was always just considered a band-aid that did nothing more than appease management and allow developers to continue writing insecure code. The thought obviously being that the application doesn't have to be perfect (or even near perfect), since the WAF will keep most of the "baddies" out anyway. WAF's were just another tool that was brought in to make up for the lack of a developer's ability to write secure code.
I do agree that multiple layers of security (WAF, IDS/IPS, SSL, secure code, etc.) when implemented properly, are extremely effective. However, the primary problem is that traditionally one security measure comes into use simply to make up for the shortcomings of another. The root of the problem with insecure software will always lead you back to the developers.
Companies who continue to invest millions in these automated tools, while additionally continuing to neglect investing money into developer training, is analogous to building a big fence around your house to prevent intruders, rather than taking the time to actually lock your doors and windows. The money being spent, in the end, continues to ignore the root of the problem.
@Mephisto, I agree with much of what you had to say, except it being a "developer" problem. Sure, a solution to more secure code flows through them, but I think more core to the problem are EULAs.
EULAs state that there is no warranty, no guarantee, and no liability. Under this business case, there is little incentive for an organization creating software to make software security a requirement. And if there is no software security requirement, we cannot possibly blame developers as a group for writing insecure code.
No amount of "developer training" is going to fix this misalignment of incentives.
Oh, and so we get as you put it, "security measure comes into use simply to make up for the shortcomings of another."
Hi Jeremiah,
I am Siddharth Narayanan, a graduate student in Information security at Carnegie Mellon. I am interested in a security position at whitehat. I am sorry at having to comment on your blog but I need to get your attention!
I believe i have decent knowledge of web and application security and would like to be considered for a position at white hat. I have already applied online but the response has not been positive. I believe I should at least be given a chance to display my abilities before being rejected. The online questionnaire does not really identify if someone has any knowledge of web security. For example, knowing what is a valid variable name in PHP (google is good enough for most questions) does not really reflect if someone understands how the same origin policy is at the crux of web security. I ask to be at least phone screened for the position before being rejected. I am grateful for your time and patience. I am a novice in the field but I am passionate about security and am looking for a good place to start my career at. I see myself solving some important problems in this area. I really hope people at whitehat will at least consider giving me an interview for this position.
Thank you,
Siddharth
@Jeremiah, I agree the organization as a whole must buy into software security. However, at this point in the game, after the TJX fiasco, Heartland Payment Systems and every other security breach that gains news headlines, I'd think all aspects of security would be a business goal. Weren't there statistics that showed that 80% of all attacks are aimed at the application level now?
At some point I'd also think consumers would start saying, "if you can't create secure applications that protect my data, I'll find another vendor who will". This in itself should be a motivating factor for businesses to require security in anything they create and provide for their customers.
@Mephisto apparently the losses, and risk of loss, haven't been high enough. Or, the losses haven't been well enough tied back to the cause (software security). In either awareness and evangelism must continue.
My argument against full reliance on WAFs and for actual development fixes is simple.
Yes a WAF can help you, but a WAF only gives two options.
1) If fails closed, your web app ends up DoSed essentially, bye bye customers, bye bye revenue/profit.
2) If fails open, you end up being attacked or taken advantage of, bye bye revenue/profit, and after the disclosure (depending what has happened) bye bye customers.
@jwilliams
"There's one waf use case that seals the deal for me. The capability to quickly deploy a 'virtual patch' is something that every organization should have in place."
This is the one true advantage I see for WAFs. Though it's hard to get others to see that this is a "virtual patch" and that actual development is still needed.
@kingthorin And so the choices for many organizations are:
1) do nothing (not fix the code or virtual patch)
2) virtual patch (risk of still not fixing the code)
While the second option is obvious not what we'd like, its still preferable to #1.
And in my experience, the vast majority of WAFs are deployed in "fail open" mode. Which is why WAF DoS attacks are especially interesting to me. :)
I think anti-waf sentiments are similar to anti-DLP sentiments, and probably share similar reasons.
To be painfully brief, and repeat may things already said:
-compliance checklists
-compliance vs security priorities
-vendor marketing doesn't help at all
-no staff, so orgs desire to buy tools
-fire-and-forget mentality to those tools
-seen as "easier" alternative
-shifts burden/cost away from devs (perceived)
-lack of understanding limitations
-lack of understanding care-and-feeding
-security mentality that tools must be perfect (DLP and wafs can't be perfect)
And so on. It doesn't help that many in security understand these things like the care-and-feeding of these tools and the limitations. Instead, they get burned by being budgeted for tools but not the staff to manage them properly. This leads to backlash...
Further, a WAF is a cross-disciplinary piece. A developer alone usually can't do it. Security alone usually can't do it. It takes security knowledge, code/app knowledge, and even networking. That's pretty advanced, especially when firms keep buying WAFs for the above reasons and don't staff/operate them with qualified people. WAFs are set up to fail, and they're not perfect, so they get into this downward spiral of anti-wafism.
A WAF is necessary, if you ask me; it sits in too-powerful a spot to be ignored by being a choke-point in front of a web app a-la load balancers. It's just an unfortunate, advanced thing compared to all the other easier stuff orgs are doing wrong.
-LonerVamp
The problem is the way you've set this up. Once you describe criticism of WAF's as "zealotry", you've poisoned the well with emotional language. Of course I'm not going to defend "zealotry". But I will defend criticism of WAFs.
Your post sets up a false dichotomy: either you're an anti-WAF zealot, or you're a WAF supporter. But there is an intermediate position. It's possible to recognize that WAFs are not completely valueless, yet also have major shortcomings -- without being a zealot.
Think back to the 90's, when a perception started to spread that all you needed to be secure was to deploy a firewall. Security folks had to educate people that firewalls are not some kind of magic that solves all security woes. The situation with WAFs is a little bit similar today. Many WAF vendors vastly overhype the technology; some customers have bought into the hype; yet in practice WAFs offer very modest value. I think it's valuable to have WAF critics to point out their shortcomings.
"...when WAFs are purchased “first,” the software security guys lose money and are resentful."
Wow, I guess they are really going to hate ryu technology then, only ryu does not suck and is more an entire Web app stack firewall, with some major differences. However, it does address the issue of protecting Web app vulnerabilities already out there, the lack of resources for code remediation and the huge shortfall of software security experts that you point out. See:
http://www.trustifier.com/ryu#compare
Post a Comment