I spend a lot of time with companies, mostly large and medium sized, who are interested in finding the vulnerabilities in their websites. Obviously the first step in the VA process is to first FIND the websites. Now this may come as a surprise to many, companies with more than 5 or 6 websites tend not to know what they are, what they do, or who’s responsible for them. And if they don’t know what websites they own, there is no hope of securing them.
Finding all of a company’s websites isn’t exactly a trivial process and doesn’t end with scanning an IP ranges for port 80 and 443. Virtual hosts, redirects, vanity hostnames/domains, partnerships, and legacy are hurdles that must be overcome. Here is a process that should help:
1) Network Discovery
Find a starting point IP address. Most of the time the main website (i.e. www.company.com) works fine. Look up the IP address using dig or some other utility:
Next plug in the IP address in the ARIN whois database to search for the registered netblock(s). Then have a chat with one of the network systems administrators, asking them if this is indeed your netblock and if they know of any more that might have been missed.
Last, nmap scan the netblock ranges on port 80 and 443 looking for web servers. Sure other web servers could be listening on non-standard ports, but those are likely out of “web application security” scope and can be addressed later.
> nmap -sT -p 80,443 x.x.x.0-255
Save all your results in a spreadsheet.
2) DNS and Zone Transfer
Search for web servers based upon domain names by interrogating the name servers. whois works great on the command line, but if not, any other website (godaddy, register.com, etc) will do that provides the service.
> whois company.com
Name Server: nameserver1.com
Name Server: nameserver2.com
Next we’ll attempt a DNS zone-transfer on the off chance that it’s misconfigured. Digital Point Solutions provides a great online utility that does this for you, which loops through each name server attempting the zone-transfer. dig on the command line works fine as well, but I still prefer the Web in this instance.
> dig @ nameserver1.com company.com axfr
> dig @ nameserver2.com company.com axfr
Additionally it doesn’t hurt to have a chat with the person in charge of or has access to the domain registrars account to see what other domain names are owned by the company. If you are lucky they might even save you a lot of work by providing the hostname list from the DNS name servers directly. If you have access to the web servers configuration or the person who does, you could also dump the virtual hostnames and get lists that way as well.
Match up the hostnames to the IP addresses in your spreadsheet and log the domain names.
3) Google and Netcraft
Google is a great resource to locate websites, especially if you know the right search options to use. First restrict search results by domain name:
This should provide a list of results, but also a lot of pages on the same hostnames that need to be widdled down. Once you find a hostname, log it, then restrict it from the search results and try again.
site:company.com -www.company.com -store.company.com
and so on until no more results come up. Log all hostnames found.
Netcraft SearchDNS is also an excellent resource for locating hostnames. Perform a wildcard domain name search for each domain name you have logged:
Log each hostname listed. You’ll probably get a lot of overlap between Google and Netcraft, but that’s OK, better not to miss anything. You also might want to give Fierce (by Rsnake) a try… it locates targets both internal and externally, not just websites though.
4) The grunt work
Visit each website on the list with a web browser and start taking notes. See if the website is up, active, functional, its purpose, redirects to or anything else informational. Click around the website, having a look at the links and the sitemap to see if any other hostnames or domain names are not on your list. Doing this with a logging HTTP proxy helps as well.
Depending on how much websites there are, this can be a painstaking process, but it’s also vital.
How to rate the value of your websites (Road to Website Security part 2)