My Attribute-Based Cross-Site Scripting post stimulated an interesting exchange in the comments. You can read for yourself, but one key thing is that this particular issue is not new. In fact it’s been around for years and only now are we figuring out how to scan for it more effectively through trial and error on real live websites. A couple of people picked up on this and one blogger asked a particular relevant question:
“If current web application scanners can't find an issue which is around for 5 years now, aren't they f*** useless?”
The frustrated tone of the question is obvious. As someone who’s been on the customer side of VA solutions, I understand where they’re coming from. We all know web application scanners don’t (and never will) find every vulnerability, but it's imperative to know what they do check for and how well. That’s the point being made. Just because a tool says it “checks” for something, it doesn’t mean it's any good at “finding” it. This is a key piece of information to efficiently complete an assessment and not wasting time overlapping work.
I’ve talked about this lack of knowledge in my web application scan-o-meter post and invited others to comment, including the scanner vendors. Their marketing teams are very good about generating a big list of “we check for this”, so I set the dials where I believed the state-of-the-art to be. Nothing really came of it from their side. My conclusion is they simply don’t know. While they can test their products in the lab, they’re unable to measure real world capabilities on any kind of scale, where their customers actually use them, like WhiteHat does. Hence their technology improvement is painfully slow, resulting in frustrated questions like the above.
The bottom line is automated scanning is important to the vulnerability assessment process. But it doesn’t help anyone when technology capabilities are withheld from customers. I’m hopeful the next sets of web application VA solution reviews will shed light on this from an independent source.