Will Netcraft Crawl "MY" Web-Site?
Netcraft have defined some requirements for web-site crawling to take place:
- You MUST be using an operating
system that is supported. Some OS' are more difficult than others to
get an uptime statistic from, e.g. Windows 98FE (unpatched) will roll
over uptime every 49 days.
- Your server, whether it be a mail-server, Secure HTTP (HTTPS)
server, or HTML Server (HTTP) must be configured with the appropriate
ports opened AND have a domain-name associated with that open port
(Such as LimestoneFormation)
- You need to use a supported Server-Software, such as Apache or Internet Information Services (IIS)
- The internet connection must be working, and not blocked by a firewall.
- Either the Netcraft Bot, or a
person visiting or accessing your server with the Netcraft Extention
Toolbar needs to use your resource for a report to be generated
- You must not block the Netcraft Crawler/Bot in your "Robots.txt," or other files that restrict access to your site