Recently we've had a problem with high trafic load on our servers for FeminaPortal - portal and forums for women, girls and ladies. It is well optimized and so far there were not any issues with it.

We determined that the problem was caused by Google search engine crawlers that performed many requests and that had impact our online classified site. It looks like we were being rapidly crawled by GoogleBot.

You may be able to reduce the effect that search engine crawlers have on your site by creating a robots.txt file. You can create one in your domain's document root with the following content: 

User-agent: *

Crawl-delay: 10

This works for most search engines, however, Google, for example, can be a little more picky. In order to adjust Google or Bing's crawl rate, you'll need to follow the steps at our blog articles:

- How to tell Google to change crawle rate

- How to tell Bing to change crawle rate

Some search engines such as Baidu, however, are entirely unfriendly and do not respect any known method to restrict them from crawling pages. For these, we recommend blocking them entirely by using .htaccess rules.