3

I'm in a difficult situation, the Baidu spider is hitting my site causing about 3Gb a day worth of bandwidth. At the same time I do business in China so don't want to just block it.

Has anyone else been in a similar situation (any spider)? Did you come across a magical solution? Or did you just accept it and either block or not block the bot?

d.lanza38
  • 407

1 Answers1

3

As long as it follows robots.txt you can throttle the requests

User-agent: *
Crawl-delay: 10
Mike
  • 22,748