A while back I published what became an extremely popular post, looking behind the scenes at WhiteHat Sentinel’s backend infrastructure. On display were massive database storage clusters, high-end virtualization chassis, super fast ethernet backplanes, fat pipes to the internet, near complete system redundancy, round-the-clock physical security, and so on. Seriously cool stuff that, at the time, was to support the 2,000 websites under WhiteHat Sentinel subscription where we performed weekly vulnerability assessments.
Today, only seven months later, that number has nearly doubled to 4,000. A level of success we’re very proud of. I guess we’re doing something right, because no one else, consultancy or SaaS provider, comes anywhere close. This is not said to brag or show off, but to underscore that scalability is a critical part of solving one of the many Web security challenges many companies face, and an area we focus on daily at WhiteHat.
To meet the demand we scaled up basically everything. Sentinel now peaks at over 800 concurrent scans, sends roughly 300 million HTTP requests per month, a subset of which are 3.85 million security checks sent each week, resulting in around 185 thousand potential vulnerabilities that our Threat Research Center (TRC) processes each day (Verified, False-Positives, and Duplicates), and collectively generate 6TBs of data per week. This system of epic proportions has taken millions in R&D and years of effort by many of the top minds in Web security to build.
Clearly Sentinel is not some off-the-shelf, toy, commercial desktop scanner. Nor is it a consultant body shop hiding behind a curtain. Sentinel is a true enterprise class vulnerability assessment platform, leveraging a vast knowledge-base of Web security intelligence.
This is important because a large number of corporations have hundreds, even thousands of websites each, that all need to be protected. Being able to achieve the aforementioned figures, without sacrificing assessment quality, requires not only seriously advanced automation technology, but development of a completely new process of performing website vulnerability assessments. As a security pro and vendor who values transparency, this process, our secret sauce, something radically different than anything else out there, deserves to be better explained.
As a basis for comparison, the typical one-off consultant assessment/pen-test is conducted by a single person using an ad hoc methodology, with one vulnerability scan, and one website at a time. Generally, high-end consultants are be capable of thoroughly assessing roughly twenty websites in a year, each a single time. An annual ratio of 20:1 (assessment to people).
To start off, our highly acclaimed and fast growing Threat Research Center is the department responsible for service delivery. At over 40 people strong, the entire team is located at WhiteHat headquarters in Santa Clara, California. All daily TRC workload is coordinated via a special software-based workflow management system, named “Console,” we purpose-built to shuttle millions of discreet tasks across hundreds/thousands of websites that need to be completed.
Work units include initial scan set-ups, configuring the ideal assessment schedule, URL rule creation, form training, security check customization, business logic flaw testing, vulnerability verification, findings review meetings, customer support, etc. Each of these work units is able to be handled by any available TRC expert, or team of experts, who specialize and are proficient in a specific area of Web security, that might take place during different stages of the assessment process. Once everything is finished, every follow-on assessment becomes automated.
That is the real paradigm buster, a technology-driven website vulnerability assessment process capable of overcoming the arcane one-person-one-assessment-at-a-time model that stifles scalability. It’s as if the efficiency of Henry Ford’s assembly line met the speed of a NASCAR pit crew — this model dramatically decreases man hours necessary per assessment, leverages the available skills of the TRC, and delivers consistently over time. No other technology can do this.
As a long time Web security pro, to see such a symphony of innovation come together is really a sight to behold. And if there is any question about quality, we expect Sentinel PE testing coverage to meet or exceed that of any consultancy anywhere in the world. That is, no vulnerability that exposes the website or users to a real risk of compromise should be missed.
Let’s get down to brass tacks. If all tasks were to be combined, a single member of TRC could effectively perform ongoing vulnerability assessments on 100 websites a year. At 100:1, Sentinel PE is 5x more efficient than the traditional consulting model. Certainly impressive, but this is an apples to oranges comparison. The “100” in the 100:1 ratio is websites NOT assessments like the earlier cited 20:1 consultant ratio. The vast majority of Sentinel customer websites receive weekly assessments, not annual one-time one-offs. So the more accurate calculation would equal 5200:1 (52 weeks). Sentinel also comes in varied flavors of coverage. SE and BE measure in at 220:1 and 400:1 websites to TRC members respectively.
The customer experience perspective
Whenever a new customer website is added to WhiteHat Sentinel, a series of assessment tasks are generated by the system and automatically delegated via a proprietary backend workflow management system — “Console.” Each task is picked up and completed by either a scanner technology component or a member of our Threat Research Center (TRC) — our team of Web security experts responsible for all service delivery.
Scanner tasks include logging-in to acquire session cookies, site crawling, locating forms that need valid data, customizing attack injections, vulnerability identification, etc. Tasks requiring some amount of hands-on work are scan tuning, vulnerability verification, custom test creation, filling out forms with valid data, business logic testing, etc. After every task has been completed and instrumented into Sentinel, a comprehensive assessment can be performed each week in a fully automated fashion, or by whatever frequency the customer preferrers. No additional manual labor is necessary unless a particular website change flags someone in the TRC.
This entire collection of tasks, all of which must be completed when a new website is added to Sentinel, is a process we call “on-boarding.” From start to finish, the full upfront on-boarding process normally takes between 1 – 3 weeks and 2 – 3 scans.
From there, there are people in the TRC purely dedicated to monitoring nearly hundreds of running scans and troubleshooting anything that looks out of place on an ongoing basis. Another team is tasked to simply verify hundreds of thousands of potential scanner flagged vulnerabilities each week such as Cross-Site Scripting, SQL Injection, Information Leakage, and dozens of others. Verified results, also known as false-positive removal, is one of the things our customers say they like best about Sentinel because it means many thousands of findings they didn’t have to waste their time on.
Yet another team’s job is to configure forms with valid data, and marking which are safe for testing. All this diversification of labor frees up time for those who are proficient in business logic flaw testing, allowing them to focus on issues such as Insufficient Authentication, Insufficient Authorization, Abuse of Functionality, and so on. Contrast everything you’ve read so far with a consultant engagement that amounts to a Word or PDF report.
At this point you may be wondering if website size and client-side technology complexity cause us any headaches. The answer is not so much anymore. Over the last seven years we’ve seen and had to adapt to just about every crazy, confusing, and just plain silly website technology implementation the Web has to offer — of which there are painfully many. Then of course we’ve had to add support for Flash, Ajax, Silverlight, JavaScript, Applets, Active X, (broken) HTML(5), CAPTCHAs, etc.
The three most important points here are:
1) Sentinel has been successfully deployed on about 99% of websites we’ve seen. 2) Multi-million page sites are handled regularly without much fanfare. 3) Most boutique consultancies assess maybe a few dozen websites each year. We call this Monday through Friday.
Any questions?
