Jobs available at Netcraft

The Netcraft Web Server Survey

Reports | Graphs | What's that site running now? | Explore
Mechanics | The First Year | Developers | Servers | Most Requested Sites

The Netcraft Web Server Survey is a survey of Web Server software usage on Internet connected computers. We collect and collate as many hostnames providing an http service as we can find, and systematically poll each one with an HTTP request for the server name.
In the July 2000 survey we received responses from 18,169,498 sites.

iPlanet is the sum of sites running iPlanet-Enterprise, Netscape-Enterprise, Netscape-FastTrack, Netscape-Commerce, Netscape-Communications, Netsite-Commerce & Netsite-Communications.

Microsoft is the sum of sites running Microsoft-Internet-Information-Server, Microsoft-IIS, Microsoft-IIS-W, Microsoft-PWS-95, & Microsoft-PWS.

Platform groupings are here.

Around the Net

Microsoft announced a new tool aimed at giving hosting companies the capability of producing Windows 2000 based web sites more quickly. The tool has been codeveloped with interland.net who are currently the largest hoster of Windows 2000 sites in the US, with around 2,500 sites. Interland also have around 30,000 sites hosted on Linux, and some 60,000 on NT4. Windows 2000 has yet to see significant takeup with hosting companies, but this was clearly set out as a goal for Microsoft by Steve Ballmer in the advent of the Windows 2000 launch.

Dogfood

Motivation

Over the last two years there has been significant growth in the internet's DNS, fueled by rife domain name speculation, falling registration prices, easier and more efficient administrative procedures, and widespread publicity. It has become more common practice for companies offering internet registration services to place a template site on the web for each domain that they register. Additionally, some hosting service companies find it convenient to create new sites at the time of customer signup, rather than at the point at which the customer is prepared to place real, personalised content on to the web.

So, whereas in the early days of the web, hostnames were a good indication of actively managed content providing information and services to the internet community, the situation now is considerably more blurred, with the web including a great deal of activity, but also some considerable quantity of sites untouched by human hand, produced automatically at the point of customer acquisition by domain registration or hosting service companies.

The biggest domain registries are large enough to be significant even in the context of the 17 million sites found by the June 2000 Web Server Survey. For example, register.com host some 1.4m domain names, the great majority of which are template sites. Network Solutions have a system hosted at Digex which hosts around 750,000 template sites for domain name holders. These two domain name registries presently account for about 12% of the hostnames found in the Web Server Survey.

Additionally, many companies will register in more than one domain. For example, Netcraft holds the netcraft.com, netcraft.net, and netcraft.co.uk domains, and currently uses three hostnames that will resolve to the Netcraft site. This means that there are nine names in the DNS that will resolve to the same content; www.netcraft.com, netcraft.com, ssl.netcraft.com, www.netcraft.net, netcraft.net, ssl.netcraft.net, www.netcraft.co.uk, netcraft.co.uk, ssl.netcraft.co.uk,

Circa 1996-7, the number of distinct IP addresses would have been a good approximation to the number of real sites, since hosting companies would typically allocate an IP address to each site with distinct content, and multiple domain names could point to the IP address being used to serve the site content.

However, with the adoption of HTTP/1.1 virtual hosting, and the availability of load balancing technology it is possible to reliably host a great many active sites on a single [or small number of] ip addresses. For example FreeServe has around 150,000 sites hosted on four load balanced IP addresses. These are substantially all active sites produced by real people crafting HTML in FrontPage, Word, Netscape, text editors, etc.

This all begs the question as to how many active sites there really are on the web.

Methodology

On IP addresses with a very large number of sites, we would generate an unreasonably large number of requests with this approach. So, the decision was taken that on IP addresses with more than 100 sites, we would sample.

The formula used to determine this number is as follows;

        N <= 100 ? N : int(100 + log10(N/100)*100);

which means that the number of pages we fetch is logarithmically proportional to the number of sites on that IP address, but will be at least 100 assuming that there are 100 or more sites on the IP address. This works out to 515 pages from futuresite.register.com's 1,414,626 sites in June 2000.

Keeping the number of requests to be under 1000 even in the most pathological case should ensure that the survey is not banned from sites, and that the results remain respectable and accurate. For the great majority of IP addresses with fewer than 100 sites, the survey is effectively a census, taking the front page from each site.

The comparison is made based on the HTML tag structure of the page, rather than the text of the whole page, as the typical scenario is that a domain registration or hosting company will use the same template tag structure for each customer, but vary some text on each site to reflect the domain name, the customer name, or the date registered, etc, which makes a strict comparison unsafe. An MD5 checksum is taken of the tag structure of each page, and this is used as the basis of the comparison, with only distinct MD5 signatures being counted.

Some companies like to use Framesets for presentation, even on inactive sites. So where the front page contains a tag, we follow the references to take the frame pages, and compare the tag structure of those in the same way as the initial front pages. Following these links is necessary otherwise there would be a risk that all front pages consisting of references would be incorrectly assessed as inactive.

The reason for doing this is that most inactive sites are automatically generated from a template, and whilst the page contents may differ between sites, the page structure remains constant.

Results

Apache loses 6,205,474 sites, causing its share to drop around three percentage points from 62.53% to 59.56%. Netscape-Enterprise loses 957,629 sites, making its share drop from 6.74% to 2.61%. Microsoft-IIS loses 1,458,421 sites, so its share of the survey rises some six and a half percentage points from 20.36% to 26.84%.

Web Server Count by Sites	Web Server Count by Active Sites

Globally, the operating system share of active web sites in June 2000 is as follows;

Operating System Count by Sites	Operating System Count by Active Sites

Compared with the by-hostname survey, Linux drops by 6%, Solaris by 4%, NT gains 7%. Other O/S's each gain by a little. On active sites Linux & NT are very close, with Linux having slightly more than a 1% lead over the combined figure for NT4 and Win2000.

As expected, a sizeable chunk of the Linux drop is at register.com which has around 1.3m inactive sites. Likewise for Solaris, Network Solutions accounts for somewhere over 800,000 inactive sites. Conversely, theplanet.net which hosts FreeServe, the largest UK dial-up ISP with around 1.3m active subscribers, only falls by about 20%, maintaining 136,000 of around 170,000 sites. This is a nice test case for the methodology, as FreeServe is a large number of active sites on just four IP addresses using http/1.1 virtual hosting.

Of the 3.3 million IP addresses with active sites, the great majority, 3.2 million, have just a single active site on them. At the other end of the scale the largest number of active sites on a single IP address is 286,620 at Webjump

Essentially, the domain name registries disappear, and the cheap or free bulk hosters come to the fore, together with the larger colocation companies such as exodus.net who may provide connectivity for "downstream" bulk hosters. This is exactly as one would predict.

Regional Variations

Conclusion

Reports and Interactive Queries

Fair Use, Copyright

Your comments and criticism are much appreciated.

Search | What's that site running now? | Most Requested Sites | Graphs
Survey | Reports | Servers | Developers | Mechanics | The First Year
Netcraft | Client List | Security | Web Business Models | SSL Survey