6 reasons why reviewing development code is not the same as assessing production websites

Finding and fixing vulnerabilities during development prior to production release helps lower software development costs and decreases website security risk. Obviously vulnerabilities in development will make their way to production, but what many people miss is that not all vulnerabilities in production once existed in development. At first this might seem counter intuitive, but it happens all the time.

Frequently WhiteHat customers deploy our Sentinel Service earlier in the SDLC (via Satellite Appliance), right around QA time. I mention this because we’ve presently run continuous assessments on a significant number of staging systems, side-by-side with the production “mirrors”. What’s interesting is that the vulnerabilities reported are rarely identical! This comes as quite a shock to many of our customers. Turns out there are a variety of reasons for differences between development/staging and production systems:

1) Code is enabled/disabled prior to release
Before staging code is released to production the push process says to disable debug == 1, admin == true, test credit card numbers, verbose error messages, test/QA accounts, etc. Then enabled extra security add-ons, transaction logging, performance enhancements, etc. This means production will never be a perfect mirror of staging and should any one step be overlooked, vulnerabilities could be introduced.

2) Lack of system configuration and patch level consistency
Even with centralized configuration and patch management, it’s extremely difficult to keep staging and production machines in perfect sync on rapidly changing websites. Non-standard-build web servers are quickly put into rotation, ASP.Net Request Validators are disabled on-the-fly because users need to upload active content, and not to mention random system failures.

3) Last minute feature requests demands
Management says a new feature needs to be pushed NOW! “Security be damned, the company will lose money if it isn’t released. Make the change on production and deal with any security issues later.” Developer’s rushes out new, buggy, and insecure code.

4) Emergency vulnerability hot fixes
Security says a high-severity-easy-to-exploit vulnerability is discovered on the production website. “Developers need to implement a hot fix ASAP because its too risky waiting for the normal dev/QA process.” Developer pushes the fix, but forgets to back port the code. The next scheduled released overwrites the earlier security hot fix.

5) Backup and log files hidden in a public available location
System administrators are trained to make backups of files (*.bak) before they’re overwritten in case something goes wrong. If backup files remain next to the original indefinitely it may never become a problem. However, if the files are located in a Web directory it can be. "login.cgi” becomes “login.cgi.bak” and the source code would be made publicly available by simply requesting “/cgi-bin/login.cgi.bak”.

6) Infrastructure devices
When it comes to infrastructure devices such as proxies, load balancers, firewalls, network storage, databases, and web application firewalls - staging systems are rarely a perfect mirror of production. Often the reason for the difference is cost. These devices have the ability to impact a websites security posture either positively or negatively.

Try as we might successfully maintaining identical staging and production systems is difficult (often impossible) to say the least. Sometimes the issues described above are under developer control, but certainly not always. Push managers, system administrators, and security personnel at the direction of the powers that be all play a role in maintaining a websites and everyone makes mistakes. This illustrates why measuring the security of the actual production website is so important.