2 options: Use all web front end computers for crawling Use a dedicated web front end computer for crawling Be careful, if you use only one webserver, do not check the option : dedicated server In Central Administration > Operations > Services on Server > Office SharePoint Server Search Service Settings, you may select one Web Front End (WFE) as the only one used by the index server during crawls. This sounds like a fine option when you have multiple WFEs in production and you want to reserve one for the index server to crawl so that the overhead is limited to one server. In a small farm, one would not normally change from the default selection above. However, on our small farm someone selected the second option and indexing failed. Error messages begin to appear in the crawl logs. When we pinged the sites to see if they were reachable from the index server, strange IP addresses responded. Since these addresses were not in DNS, we examined the HOSTS file. This revealed a undocumented SharePoint Search process. What happened, you ask? First, let’s discuss the logic. If you have multiple WFEs and you want to dedicate one of them for crawling, obviously the index server must be able to find the appropriate web sites on the dedicated WFE because the UI only specifies the farm member name. One might think that it would simply use DNS. However, DNS will resolve the target web sites to the production addresses. So a process was added to modify the HOSTS file on the index server which adds an entry for each web site to be crawled using the IP address of the WFE selected. The entries will look like this: 10.16.x.x admincompanion.mindsharp.com # Added by Office SharePoint Server Search (1/18/2007 12:43 PM). With this entry in place, the index server knows how to reach the sites on the dedicated server. However, depending upon how you have the WFE configured, this may break the crawling process! Obviously if the entry is incorrect, the crawler cannot find the site to crawl and you will see error messages in the logs saying the site is unavailable. After three attempts, it is removed from the list. If you remove the entries from the HOSTS file manually, they will be back in just a few minutes. If you correct them, your entries will be removed and replaced with the original entries. The process does not overwrite entries for non-SharePoint sites. Why might the entries be incorrect, you ask? Well, empirical tests show that the address selected by the process is the first address displayed on the first NIC displayed when you do an IPCONFIG /ALL. Also, there is one entry for each site listed in a crawl rule. Remove the site from the list and the HOSTS entry will magically disappear. Our WFEs have more than one NIC so that SQL traffic has its own uncongested pathway. Also, the front facing NIC has multiple addresses with many of them bound to SSL web sites. In our case, the first NIC was the one on the network to the SQL server and the index server could not reach that isolated network. By renaming the NICs so that the order was changed, the process selected the first address displayed on another NIC. However, that did not solve the issue because the address selected many of the sites were using SSL with bound IP addresses plus the address selected was bound to a non-SharePoint site so none of the host header sites were listening on that address. The index server could not find the web sites. So, this “feature” would be a wonderful “solution”, IF : The WFE only has one network interface. The one network interface only has one IP address or at least does not have any IP addressed bound to a particular web site. The WFE only hosts SharePoint sites which are all using [All unassigned] and host headers . We have forwarded information for a Design Change Request. However, until there is a UI to select the IP address that the process uses in its entries and/or a UI to disable this process so that manual updates to the HOSTS file will remain, you may want to consider your WFE design before making the decision to dedicate a WFE for indexing. But the real solution is. . . If you want your index server to crawl a particular WFE, leave the default setting alone and modify the HOSTS file yourself. This way you can direct it to hit one server for some sites and another for others.