Friday, May 11, 2007

Chasing Vulnerabilities for Fun and Profit II

This post comes via our Spring 2007 newsletter which also includes various industry articles and events/conferences we'll be attending. Full download.

At WhiteHat Security we spend our time hacking the world’s largest and most popular websites (a really cool job). The issues we uncover, if exploited by an attacker first, could easily result in huge financial losses or devastating consequences for a consumer brand. Thousands of these issues can be chalked up to Cross-Site Scripting (XSS) and SQL Injection, but the most technically fascinating discoveries are flaws within application business logic. These discoveries include high-severity vulnerabilities through which we could purchase laptops for a dollar, see other customers’ order information, liquidate bank accounts, and more. These issues are interesting because no two are exactly, alike and challenging because they are often so complex that they must be found through expert analysis. And, as I wrote in “Chasing Vulnerabilities for Fun and Profit I,” as a spontaneous office activity a few of us race to find the first and best vulnerability in a never-before-seen website. Special praise is given to those who find the clever logical flaws.

More often than not, the first medium severity vulnerability is identified inside of two minutes if it is cross-site scripting, and 20 minutes if it is a high severity issue. The best and most competitive races occur when the website is relatively secure and no easy SQL Injection is present. These races last longer and the results become more interesting. To win, the WhiteHat Security Operations Team has to find a business logic flaw by determining what functionality is supposed to be accessible by a user and then trying to find holes that allow him to subvert the application design. The results depend on the target website and its set of functionality.

For example, the target on a particular day was a large e-commerce application service provider (ASP). The ASP provides a technology platform to sell shrink-wrapped software. Let’s say that a customer wants to buy a particular piece of software. He would click “buy” on the software vendor’s website and then be transported to the ASP’s website to complete the transaction. The ASP’s website is “skinned” identically to the software vendor’s in order to maintain brand identity. After the purchase, the customer downloads his application and license key, or may alternatively request that a CD be shipped to him.

Hundreds, perhaps thousands, of software vendors depend on this system for order processing. Imagine all of the data that the ASP is protecting. To buy software, a customer must provide his name, address, telephone number, credit card information, and expiration date. Plus, the ASP stores software license keys. This is all highly valuable data for identity thieves and software pirates. A compromise of this data would not only be devastating to the ASP, but also to all of the software vendors that depend on the systems. As one might expect, our goal was to see if we could compromise this data through the web application. Better that we discover the flaws before attackers exploit them.

The ASP informed us that they had taken the usual security precautions including SSL, firewalls, and patched systems. Their Web application security program called for annual Web application vulnerability assessments during which holes are identified and reports generated. The developers and security staff resolved the issues within a reasonable amount of time and business continued. As is common in most e-commerce operations, the security impact of weekly code updates was an unknown. With every new line of code, the risk of introducing new vulnerabilities increased, as did the possibility to jeopardize the entire system. Also, the accuracy of last assessment report decayed each week after its completion until it became useless.

At the time WhiteHat stepped in, the ASP was about 90 days and 12 code pushes since its last assessment--more than enough time for new issues to be introduced, and perhaps a few missed issues from the previous assessment to come to light. With the team’s browsers ready to go, the starting URL was called out and the speed typing commenced. Everyone was feverishly going for the quick XSS win. In usual fashion, everyone went after the search box, but to no avail. Someone at the ASP had found and closed that issue long ago. Any place that echoes user-supplied data will do.

At roughly 30 seconds in, Theodore “T.C” Niedzialkowski (Senior WhiteHat Security Engineer), with fists in the air, shouted “scripting!,” as in Cross-Site Scripting. T.C. was quick to the spot that has to be the second most common location for XSS, the “Not Found” page. Many 404 handlers do not properly HTML encode the URL before echoing a user’s request was not found. With the first contest over, Bill Pennington (WhiteHat vice president of services) and I, not wanting to be shutout, began the hunt for a high severity issue.

WhiteHat’s Security Operations Team is NOT comprised of the Swordfish movie, Hollywood-style hackers who hack by 3D Auto-Cad. We are not the types to resort to script kiddie tools. The fact of the matter is that most scanners simply do not work fast enough on custom websites anyway, and they absolutely cannot identify complex logical flaws, at least not without massive upfront configuration. And, they certainly will not provide results within our timeframe of a few minutes. Most of the time someone will claim victory using Firefox, an HTTP proxy, and their gray matter.

Five minutes into the hunt the silence became eerie. No one was saying a word. Instead, we were all immersed in concentration, which meant that no one had yet hit pay dirt. Several of us were finding easy Information Leakage issues through which the system revealed internal paths and software version numbers in error messages and HTML comments. These are issues worth disclosing, but certainly not high severity issues. We tried testing credit card numbers, flipping product IDs, and swapping template parameter values, but nothing was working. There was no login to the system either, just a simple sessioning system for product purchasing. Not much to work with, and I was running out of application functionality real estate. I would have to start tracking back over my work. Someone had done good work on this system.

Suddenly, Bill cried “owned” – slang in our world for compromising a website. The rest of the team begrudgingly stopped typing and huddled around his laptop to see what he had found. He showed us how he was able to jump into different software vendor stores and access any customer order, complete with identify information and serial numbers. Game over. No doubt. But, the question was how he was able to do this.

The issue turned out to be one those complex and multi-layered business logic flaws. The interface to the Web application was called in by a URL parameter called “template.” The value of each template had a numeric ID. I came across this and cycled through a few numbers, but nothing interesting came up. Bill did a fairly exhaustive rotation and one particular number turned up something revealing, a page entitled “customer order service center.” On this particular page, there was a search form with an entry for an order number. Apparently, this was the interface that customer support uses to track orders when there is a problem. With a valid order number, we could see if it would bring up the number.

Bill then headed over to the store interface and started the purchase process. He checked his cookies and inside there was an “order” parameter with a long numeric string. He then pasted the string back into the search box, but the search turned up empty, likely because he had not actually bought the items yet. So, Bill deleted his cookies and entered the order process again to get a new number and continued this process a few more times. By looking at five order numbers, we could see that the numbers would increment, but not exactly sequentially. In the summary meeting, we were told that the issue must have been in the software for years and never uncovered by anyone.

In under ten minutes, we uncovered a handful of vulnerabilities from low to high severity. T.C. took the speed hack prize, Bill the best hack, and I struck out. On any given day, any member our Security Operations Team can take the prize. WhiteHat’s website vulnerability assessment methodology allows us to combine our experts’ strengths with the findings of our proprietary scanning technology to ensure that we provide timely, comprehensive vulnerability coverage of our customers’ sites, with the lowest possible false positives.

It is important to understand that it is not enough to find a handful of vulnerabilities. Hackers only need one small hole in order to get in and wreak havoc on a business. The security professional needs to be confident that he has a comprehensive and up-to-date overview of his website’s vulnerability status at any time. Most Web application vulnerability assessments require thousands of tests. So, an entirely manual process, while comprehensive, is neither practical nor desirable. Nor is a purely automated process going to work because scanners can only test for about half of all the vulnerability issues present in a website and completely miss critical business logic flaws. Therefore, the best approach to Web application security vulnerability assessment is an iterative solution that combines the best of both worlds, the automation of a scanner and the experience of an expert. The results are both timely and comprehensive. When we are not racing, this is what we do for a living. Every day.


Jordan said...

And here I was thinking the contest at RSA was nothing like the real world in the timed setup and fast pace -- sounds pretty similar though! Of course, I know there's a lot more that goes into what you guys do, but at least one part of the real world matches. ;-)

Those speed rounds sound like a lot of fun.

Jeremiah Grossman said...

ahaha yah. You'd probably do really well at those here. We haven't had one in several weeks. Too busy. :)

You know. Speed hacking would probably be a cool exercise for local OWASP/WASC meetings. Have some people bring in VMWared websites, set-up a local LAN, and some kind of prize to the winner. Might be cool!

Jordan said...

That is a good idea. And as you said before, anybody can win since there's so many places to look and you never know what the right track is going to be. Lots of fun hunting though, and by their very nature fast so easy to do at a meeting.

It would be cool to require that people who submit an app make it publicly available online, maybe as a part of some OWASP repository -- could build a really large collection of test apps to practice and learn on.

I know there's a lot of resources out there like that, but I don't know that I've seen them aggregated very well anywhere.

Anonymous said...

It's a cool idea having a race to discover vulnerabilities, but I can't help wondering who's going to write the final report :) .

When the fun is gone, putting together a valuable report is _so_ boring (yes, i get to write the reports :)
office desk vs. field ops.

mbrisby said...

Very interesting post, Jeremiah. This story sounds like a good argument for having the public and administrative interfaces hosted in two completely separate places.

Anonymous said...

Awhhh Swordfish made me do all this! ^^

Ronald van den Heetkamp

Jeremiah Grossman said...

Jordan, I think we might actually have something here. Something new and exciting to do at a local meeting rather than the standard 2 hours of slideware. I might have to try this out in the Bay Area and see how it goes.

Unknown said...

Haha speed hacks + swordfish certainly yields interesting results...thankfully you didn't quite go there in the article outright! :)

That technique really does sound fun, enlightening, and a good practice for you and your team. Keeps things fun but productive and likely builds skill in everyone. Besides, how valuable could it be to have some cocky developer say, "go ahead and try our site out a few minutes and see what you ge..." "Done!" :)

Jeremiah Grossman said...

Yah definitely. Its especially fun on real world websites as well. There simply is no substitute.

Anonymous said...

I've never seen someone get so excited about what amounts to post facto QA work before...

Jeremiah Grossman said...

Why not? Everyone in tech likes hacking stuff. At least I used to think so.

Anonymous said...

Wow Im jealous, you have the 'dream job' if you ask me.

Jeremiah Grossman said...

If your into cutting-edge webappsec, its a hard job to top thats for sure.