Google has advanced algorithms for choosing when to crawl a website. The Google search engine tries to crawl pages from a site without impact on the site's performance.

In some cases Google can affect your site crawling too meny pages in a very short period of time. In that cases you would like to tell Google to reduce crawl process because too many requests could overhelm your site, especially if you have busy site or it is hosted on shared host.

Important: Google search engine does not consider robots.txt file at all!

How to tell Google how often to crawl website

Crawl rate represents the time used by Googlebot to crawl a website and Google has a feature for adjusting the crawl rate. However, it is not possible to specifiy different rates for different parts of the site. For example, it is possible to specify the crawl rate for www.website.com and subdomain.website.com but it is not possible to change it for www.website.com/subfolder.

If you change the crawl rate it will not affect how often Googlebot visits your site, it will only change the speed of requests during the crawl process.

In order to change Google's crawl rate follow the steps:

1) Login to Google Webmaster Tools

2) If you haven't added your site to Google Webmaster Tools, add it

3) Click on your website on the Webmaster Tools Home page

4) Click Settings under Site Configuration

5) Select the desired option in the Crawl Rate

6) Consider that the new crawl rate will be valid for 90 days only