Robert Auger (cgisecurity) has posted a good write-up on the challenges faced by automated web application security assessment tools. I blogged something similar a couple months ago. Robert, until very recently, used to work for a company who develops one of the commercial and widely known web application vulnerability scanners on the market. So he someone who has good insight and knows what he's talking about."There are many challenges that web application security scanners face that are widely known within the industry however may not be so obvious to someone evaluating a product. For starters if you think you can just download, install, and run a product against any site and get a report outlining all of its risks you'd be probably be wrong."
read on...
13 comments:
Jeremiah,
The last remark in this post (about the fact that Bob used to work for a scanner vendor), clearly presents the agenda you are trying to push here...
Very populistic of you.
I have my opinions, beliefs, and thoughts which I express openly here. But before I go assuming, what "agenda" did you mean?
Well,
First of all, I personally think that most of the points that Bob mentioned in his write-up have already been solved by automated scanners, or at least by the scanners that I personally know.
JavaScript execution, complex session management, non-standard URLs, even privilege escalation, these are all things that modern web application scanners can deal with, at least to some extent.
The real complex issues that require human interaction, were not really covered fully in this write-up - and that brings us to the "logical vulnerabilities".
BTW - even scanner vendors keep saying that you can't solve the problem by only using a tool, it requires a person to drive it. To research the issues more in depth, to deal with the flow/logic problems.
===
Back to the original subject -
Claiming that Bob knows more than others about the limitations of web scanners, just because he worked for a scanner vendor, is IMHO populistic, especially since Bob didn't say anything that people didn't already know. He didn't shed any light as to the internal problems he had to face when working on the scanner. He just mentioned a bunch of things, that people outside of his company know anyway...
I think your comment about Bob's old workplace has nothing to do with what he wrote. There was no "internal knowledge" in his words.
I didn't see any "agenda" claimed, so I'll move past that part.
> First of all, I personally think that most of the points that Bob mentioned in his write-up have already been solved by automated scanners, or at least by the scanners that I personally know.
I disagree, but I'll get to that...
> JavaScript execution, complex session management, non-standard URLs, even privilege escalation, these are all things that modern web application scanners can deal with, at least to some extent.
"to SOME extent", thats the key phrase. I agree most of these problems can be solved some of the time and some website, or even in a lab. In the real world, its a whole other story. I think that was Robert's point when he said:
"In defense of these vendors they can't possibly know every single situation that can occur since site behaviors aren't strictly defined as industry standards, and are often improvised for each site's unique needs. Good tools should allow you to configure various options so that you can properly adjust them for your site."
He wasn't saying these issues were impossible or unsolvable, just hard, and have to be taken into account.
> The real complex issues that require human interaction, were not really covered fully in this write-up - and that brings us to the "logical vulnerabilities".
Your right, he didn't. I invite you to write about that aspect so more people understand it.
> BTW - even scanner vendors keep saying that you can't solve the problem by only using a tool, it requires a person to drive it. To research the issues more in depth, to deal with the flow/logic problems.
Can you show me on one scanner vendor website, maybe your employers, where it says that?
===
Back to the original subject -
> Claiming that Bob knows more than others about the limitations of web scanners, just because he worked for a scanner vendor, is IMHO populistic, especially since Bob didn't say anything that people didn't already know.
I didn't say that Robert knew more than others, I said "he someone who has good insight and knows what he's talking about." Others may fit the same profile. What I do know that in his capacity he was in a good position to know. IMHO, those who perform QA, know product capabiliy better than anyone.
> He didn't shed any light as to the internal problems he had to face when working on the scanner.
Im sure he would have liked to do so, but probably restricted by NDA.
> He just mentioned a bunch of things, that people outside of his company know anyway...
I know, you know, Bob knows, the experts in the field know. But the other 99% of the people in the web application security space do not. They just assume press "go". Tell me thats not the typical behavior or a scanner customer.
>I think your comment about Bob's old workplace has nothing to do with what he wrote. There was no "internal knowledge" in his words.
I don't know if there was supposed to be. He was speaking to the customer and what they must take into consideration. If these problem have already been solved, by you or something else, then I invite you to reveal it. Inquiring minds want to know. But your going to have to do better than simply telling the rest of us "no, your wrong, these have been solved!", without sharing how.
With regards to your agenda, I think it is obvious...I am not going to explicitly talk about it.
Regarding the difference between manual testing and automated testing, Danny Allan (Watchfire), wrote a nice document about this subject. You can find it here: https://www.watchfire.com/securearea/whitepapers.aspx?id=21
As far as I know (and as far as I can help it) my company never claimed to automate 100% of the process, nor do we preach that our product will fix all of your problems by the click of a button. I personally try to fight this kind of attitude.
A final note about the next generation of scanners...I can’t speak on behalf of other vendors, but I can definitely say that Watchfire’s AppScan has gone a long way in the past few years, and the upcoming version includes several solutions that try to improve/solve some of the problems that Bob mentioned. In addition, we continue to research and develop new solutions and new ways to solve some of those hard problems.
Hey!
A few things
* Due to NDA I can't talk about certain things at my previous employer. I can't release anything about upcoming projects or internal research. Regarding your statement on me not 'shedding light' on how something is solved this would be proprietary information and a violation of my NDA.
* Jeremiah was just stating that I was in a good position to know about the issues (having done QA/r&d) and not the definitive person on the subject.
* This article is geared towards customers not people in the industry. As I stated this is known within the industry however may not be known to the general public. Here's a quote of this statement
"There are many challenges that web application security scanners face that are widely known within the industry however may not be so obvious to someone evaluating a product."
* Another point that I figured the industry would appreciate (and I have had others contact me about it) is in regards to expectations of these tools. You can't just aim and fire and while no vendor claims that you can, the general public (based off of my experience speaking with people, reading mailing lists, etc) still thinks this is the case. I merely wanted the public to know some manual work is always required and to not continue with this mindset. I also pointed out that good tools will allow configuration of some of the items that I've listed. This is a general statement not aimed at a particular solution.
This article never mentioned consultants or specific solutions and therefore was fairly neutral (and I don't work for a vendor so there is no agenda here).
* If WF can solve many of the issues and this is old news to you, you may want to consider sharing this publicly somehow. I'm not aware of it and I'm sure some customers are also not aware of it. Again this article is a general statement of these sorts of products and never says that any particular issue can't be solved, just difficult.
Let me know if you have any questions
- Robert
Bob,
I didn't express any issues with regards to your write-up. I merely stated that IMHO, Jeremiah's remark was a bit populistic.
It seems that Jeremiah fails to see my point, so I will say ciao for now...
> With regards to your agenda, I think it is obvious...I am not going to explicitly talk about it.
OK, if you say so. Personally I don't like being accused of having a hidden agenda. Any agenda I have is well documented and I make it clear where I'm coming from in post after post. Revealing my biases, employer, opinions, and thoughts out in the open.
> Regarding the difference between manual testing and automated testing, Danny Allan (Watchfire), wrote a nice document about this subject. You can find it here: https://www.watchfire.com/securearea/whitepapers.aspx?id=21
So your telling me before a customer is informed of the issues of what your scanner and can't find, they have to register for a white paper? Nowhere on the site is it laid out? No need to answer these.
> As far as I know (and as far as I can help it) my company never claimed to automate 100% of the process, nor do we preach that our product will fix all of your problems by the click of a button. I personally try to fight this kind of attitude.
That's exactly what we're talking about here. What you, I and Robert understand and fight against in not whats being claimed by the sales/marketing people who customer interact with. And don't tell me the majority people know what we do, and this is old news, because they they don't.
I plan on writing much more on the subject because I believe this subject is still not well understood, even by the experts.
> A final note about the next generation of scanners...I can’t speak on behalf of other vendors, but I can definitely say that Watchfire’s AppScan has gone a long way in the past few years, and the upcoming version includes several solutions that try to improve/solve some of the problems that Bob mentioned. In addition, we continue to research and develop new solutions and new ways to solve some of those hard problems.
So again, your telling us the issues are solved in a yet to be released product which we can't test or measure or know whats going on. In fact I bet its even against the license agreement for users to do reviews, reverse engineer, or anything else to further understanding.
I realize many of these decisions are likely out of your control, thats fine, but thats something the rest of everyone else needs to know as well.
>So again, your telling us the >issues are solved in a yet to be >released product which we can't >test or measure or know whats >going on.
I didn't say that ALL issues are solved, I said that we put an effort to solve SOME of these issues that Bob mentioned.
The GA date of the next version is very close, and is published on our web site. :-)
Ciao
I would just like to say my piece about the subject. WEBINSPECT ROCKS!
;)
Caleb
AHAAH, uh oh, this is not going to bode well for future comments. :)
I for one welcome our new scanning overlords.
Hey Caleb,
You missed all the fun ;-)
Post a Comment