Thursday, June 19, 2008

Ultimate Scanner Accuracy Test

Imperva issued a press release and a pair of blog posts describing their new SecureSphere (WAF) API functionality (OpenSphere). Cool stuff. Clearly from the text they’re excited by the VA+WAF concept, which is the same technology integration path WhiteHat Security completed with F5’s ASM and Breach’s open source ModSecurty. We’ve been getting email asking when we plan to integrate Sentinel with Imperva (and we likely will) because we’re only VA scanning vendor not listed (Cenzic, HP, IBM, NTO) in their announcement, but I’ll get to that in a moment.

There is something else deep underneath that VA+WAF brings to the surface, something everyone finds important, and that is true scanner accuracy. For VA+WAF to work (block) in a production environment scanner results MUST be EXTREMELY accurate. We’re talking sub 1% false positive and duplicate rates or things will fail (badly). And even then you still want to test virtual patches in alert mode first. Scanners cannot fudge these results, use noncommittal “potential vulnerability” reporting, or use clever marketing spin. You got the good or you don’t and probably an interesting new way to review scanners.

For example Kavado (defunct) attempted the VA+WAF in 2002-2003; and other vendors tried again in 2003-2004 using AVDL. Ultimately, all VA+WAF attempts proved unsuccessful because the scanning products dumped hundreds or even thousands of false positives and duplicate vulnerabilities (still the case?) into the WAFs. Early implementations labored WAF performance or rendered the entire website inaccessible. We feel at WhiteHat Security we’ve overcome the accuracy challenge and statistically gathered enough proof points to feel comfortable bringing the solution to market.

As far as integration our plans go and why we’re not listed, WhiteHat Sentinel features are almost entirely customer demand driven and WAF integration is no different. Many enterprises are considering their options (Cisco, Citrix, WebDefend, etc.) and then let us know which product they liked best. Then we get the product in the lab, integrate the technology, test, test, test, and test again, then show it off to a several people. If feedback is positive, then we announce. Perhaps other scanning vendors do it differently and don’t mind marketing vapor-ware. We prefer offering solutions that work first and that might explain why we’re not on the list. :)

4 comments:

Rafal Los said...

Jeremiah - readers...

Let's set a few things straight:
1) No one with any reasonable credibility will tell you a scannning tool has a 0% false-positive/false-negative rate
2) WAFs are and will never be a solve-all, permanent solution

That being said - I feel it important to mention that Scanner tools are always being perfected and pushed closer and closer to the zero-margin of error - but again, I seriously doubt it'll ever get there. A WAF/VA combination will still inherently be flawed because the business typically has a lower tolerance for "security-caused issues" than actual vulnerability-based risk... I know it's sad but that's the truth and reality. It's admirable to work towards an "ideal solution" with tools and hardware detection, etc... but let's be realistic.

* The only solution to web-based application security is to write inherently secure code* Period, end of story.

Jeremiah Grossman said...

@rafal, OK I'm with ya, but you didn't answer the most important question. Sure the FP/FN rate can never be zero, but what is it? Or, what is it claimed by the vendors? They say is "low", but cmon now, relative to what.

You also said, "let's be realistic". I don't think its realistic to believe that all inherently insecure code will be rewritten, or even close to it for matters of time/cost. Do you?

Anonymous said...

Hi Jeremiah,

Quote: "Attempts proved unsuccessful because the scanning products dumped hundreds or even thousands of false positives" -

From my experience and knowledge, this was hardly why the integration failed -

IMHO - the WAF/VA integration idea never took off the ground, because (historically), no one saw the actual added value. It always seemed nice on paper, but in reality, no one seemed to really care about such integration.

Add to that, the low adoption rate of WAFs in the 1997-2006 period, and you understand why people were never crazy on this idea.

An additional point is that back in the day, WAFs also boasted smart positive-security engines, which theoretically did not need signatures - if a WAF vendor even mentioned attack signatures, he would immediately get stoned (by rival sales & marketing teams - you like these people, eh?) for admitting its product uses what then was considered "negative security".

Things have changed, WAFs are more common, VAs are more common, and the integration between them seems like a natural course of action (I personally think that this integration is taking place because everybody is bored, and are looking for something to do, but hey - that's just me).

-Ory

ae said...

@Ory -- I'm going to summarize/paraphrase your points and address them

Hi Jeremiah,

Quote: "Attempts proved unsuccessful because the scanning products dumped hundreds or even thousands of false positives" -

From my experience and knowledge, this was hardly why the integration failed -


Actually, early on, I know this was the case. I know several places that kicked out the Netcontinuum & SPI integration for this very reason. And the same with Kavado -- plus the fact their technology just didn't work well. I didn't experience any Appscan + WAF integration, but Appscan suffered from the same artifacts that caused this failure (e.g. - 30,000 duplicates/attack vectors flagged as unique vulns for XSS on one domain).

IMHO - the WAF/VA integration idea never took off the ground, because (historically), no one saw the actual added value.

No doubt there's some truth to this too. However, I was working with a few large back-end Financial Services providers who actually wanted to do this clear back in 2002-2004 and nobody could make their WAF+Scanner technology work.

An additional point is that back in the day, WAFs also boasted smart positive-security engines, which theoretically did not need signatures - if a WAF vendor even mentioned attack signatures, he would immediately get stoned (by rival sales & marketing teams - you like these people, eh?) for admitting its product uses what then was considered "negative security".

This still remains a problem today. A number of WAF vendors (and their sales people) still push ridiculous stories to their prospects. One large Sentinel client was recently told by a leading WAF vendor that their WAF could be up and running in front of ALL of their applications in under 2 hours.

The implication was that it would be doing more than "just plugged in". They kept talking about what you get out of the box, and how it learns and keeps getting better.

I know this particular WAF, and it is very weak on Attack Vector detection, about all it does out of the box is some protocol protection (no "POST to GET" or HTTP BoFs). Anyway, when pressed for specifics about protecting specific vulns known in the prospects software, the vendor reps danced around and talked about positive security policies you could create, if needed.

WAF marketing is still largely pushing security pixie dust aka magic-box with a magic-elf inside auto-doing everything for you. Didn't every product segment, FW, IDS, IPS, HIDS, SEMS, go through this silliness much faster?

Cudos to F5 for breaking the mold here.

(I personally think that this integration is taking place because everybody is bored, and are looking for something to do, but hey - that's just me). -Ory

Nah. WhiteHat is doing it for the most legitimate reason of all:

Our customers. They ask for & want this, and flat-out have a need to operationalize mitigating their vulns.

It takes people a long time to fix things, and the bigger you are, the more sites you have, usually the more distributed and disparate your dev teams are, add in regional/political/language barriers, and you aren't going to solve things quickly with any SDLC sauce.

We have the industry's first true platform for webappsec vuln detection and response. We enable people to take concise, actionable security data and turn it into a first-response by enabling attack vector detection or blocking that can be fully operationalized (e.g. not require dev resources).

Well, this is too long so I'm done.