I've started a relatively new website, and I submitted it to google and everything. I use google's webmaster tools as well. I'm wondering how to figure out the frequency of google's spider accessing my website. I always hear people talking in forums about "I'm getting 250 pages crawled an hour" and so on... Is there software to use? A javascript method similar to google analytics? Or is there something to analyze my server's logs? Thanks
Asked
Active
Viewed 2,944 times
8 Answers
4
Just check the webserver logs if the visitor's User-Agent request header matches/contains Googlebot. There are lot of webserver log analyzer tools, either free or payware. Most of them are also able to categorize bots. From them all I've had the best experience with Google Analytics.
2
Signup for Google Webmaster Central, aka Webmasters Tools, verify your site, and wait for the stats to appear.
Pentium10
- 454
- 1
- 9
- 24
1
It's fairly unlikely Google Bot will run any Javascript on your site that will track it arriving, so the best hope is to look at the site logs themselves - either in webalizer or the hundreds of other log file parsers, or just open them up in a texteditor and run a regex to extract the Google Bot user agent.
You should see then how many hits it makes in a space of time. If it's too much you might want to look at using an XML sitemap
Chris S
- 113
1
AWstats is a good, free utility to use for log file analysis. It'll give you a lot of interesting statistics in addition to the Google crawl info.
Mahmoud Al-Qudsi
- 618
0
I would recommend Google analytics, it covers all visits to your site and you can work out whether a visitor was a bot or not. It is also excellent for SEO on your site helping to increase visitors.
Burt
- 160
0
You can use googles webmaster tools, It will tell you how often the bot has crawled your site...
Jacob
- 9,282