I always asked myself how fast web crawlers are and how much time takes to have a visit from a bot when you add new content to your site?

This question popped up again when some minutes ago I was surprised with what my notebook screen showed to me :O (I have an open screen 24/7 showing me the logs of my web server to see the live traffic going to my site – Yes I am a freaky…). In the following log you will see that when I finished with the post of the How to take screen shots of your Blackberry device a hungry for data Google Bot has eaten my new post after 8 seconds of post life, YES ONLY 8 SECONDS!

96.30.25.56 – - [14/Sep/2009:18:24:06 -0300] “GET /wp-content/themes/index.php HTTP/1.1″ 200 – “-” “PHP/5.2.9″
96.30.25.56 – - [14/Sep/2009:18:24:06 -0300] “POST /wp-cron.php?doing_wp_cron HTTP/1.0″ 200 – “-” “WordPress/2.8.4; http://www.andresmontalban.com”
96.30.25.56 – - [14/Sep/2009:18:24:07 -0300] “GET /wp-content/themes/index.php HTTP/1.1″ 200 – “-” “PHP/5.2.9″
206.53.153.222 – - [14/Sep/2009:18:24:06 -0300] “POST /xmlrpc.php HTTP/1.0″ 200 163 “-” “wp-blackberry/0.9.0.149″
96.30.25.56 – - [14/Sep/2009:18:24:10 -0300] “GET /wp-content/themes/index.php HTTP/1.1″ 200 – “-” “PHP/5.2.9″
206.53.153.140 – - [14/Sep/2009:18:24:09 -0300] “POST /xmlrpc.php HTTP/1.0″ 200 157 “-” “wp-blackberry/0.9.0.149″
96.30.25.56 – - [14/Sep/2009:18:24:17 -0300] “GET /wp-content/themes/index.php HTTP/1.1″ 200 – “-” “PHP/5.2.9″
66.249.71.72 – - [14/Sep/2009:18:24:17 -0300] “GET /feed/atom/ HTTP/1.1″ 200 57230 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
96.30.25.56 – - [14/Sep/2009:18:24:18 -0300] “GET /wp-content/themes/index.php HTTP/1.1″ 200 – “-” “PHP/5.2.9″
66.249.71.72 – - [14/Sep/2009:18:24:18 -0300] “GET / HTTP/1.1″ 200 70820 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
96.30.25.56 – - [14/Sep/2009:18:24:45 -0300] “GET /wp-content/themes/index.php HTTP/1.1″ 200 – “-” “PHP/5.2.9″
66.249.71.72 – - [14/Sep/2009:18:24:44 -0300] “GET /how-to-take-screen-shots-of-your-blackberry-device/ HTTP/1.1″ 200 25415 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

This made me wonder if this was just coincidence or if it was a predictive visit of Google web crawler to my site, does anyone have an idea of the average time between visits of the Google web crawler bot?

I tried to do some research about this subject but didn’t found any reliable information or resource. Now I will post this article and will take the time that takes to have a visit from Google web crawler

Will keep you posted…

Update:

Well this is very interesting… As you can see in the following log I posted this article at 20:45:51 GMT-0300 and I got the first visit of the Google bot at 20:46:26 GMT-0300 that is 35 seconds!

I am wondering if this is a good practice or it is just that Google loves my blog hahahaha. Here is the new log:

190.135.11.84 – - [14/Sep/2009:20:45:48 -0300] “GET /wp-admin/post.php?action=edit&post=125&message=6 HTTP/1.1″ 200 53315 “http://www.andresmontalban.com/wp-admin/post-new.php” “Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.2) Gecko/20090803 Linux Mint/7 (Gloria) Firefox/3.0.6, Ant.com Toolbar 1.2″
190.135.11.84 – - [14/Sep/2009:20:45:51 -0300] “GET /wp-admin/load-scripts.php?c=1&load=hoverIntent,common,jquery-color,suggest,wp-ajax-response,wp-lists,jquery-ui-core,jquery-ui-sortable,postbox,slug,post,thickbox,media-upload,word-count,jquery-ui-resizable,admin-comments,schedule,autosave&ver=9af98cd7d781478365ef3128661bfbd2 HTTP/1.1″ 200 37916 “http://www.andresmontalban.com/wp-admin/post.php?action=edit&post=125&message=6″ “Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.2) Gecko/20090803 Linux Mint/7 (Gloria) Firefox/3.0.6, Ant.com Toolbar 1.2″
96.30.25.56 – - [14/Sep/2009:20:45:49 -0300] “POST /wp-cron.php?doing_wp_cron HTTP/1.0″ 200 – “-” “WordPress/2.8.4; http://www.andresmontalban.com”
190.135.11.84 – - [14/Sep/2009:20:45:52 -0300] “GET /wp-includes/js/tinymce/wp-tinymce.php?c=1&ver=3241-1141 HTTP/1.1″ 200 79103 “http://www.andresmontalban.com/wp-admin/post.php?action=edit&post=125&message=6″ “Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.2) Gecko/20090803 Linux Mint/7 (Gloria) Firefox/3.0.6, Ant.com Toolbar 1.2″
96.30.25.56 – - [14/Sep/2009:20:46:26 -0300] “GET /wp-content/themes/index.php HTTP/1.1″ 200 – “-” “PHP/5.2.9″
66.249.71.72 – - [14/Sep/2009:20:46:26 -0300] “GET /feed/atom/ HTTP/1.1″ 200 65706 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
96.30.25.56 – - [14/Sep/2009:20:46:27 -0300] “GET /wp-content/themes/index.php HTTP/1.1″ 200 – “-” “PHP/5.2.9″
66.249.71.72 – - [14/Sep/2009:20:46:27 -0300] “GET / HTTP/1.1″ 200 80492 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
96.30.25.56 – - [14/Sep/2009:20:46:56 -0300] “GET /wp-content/themes/index.php HTTP/1.1″ 200 – “-” “PHP/5.2.9″
190.135.11.84 – - [14/Sep/2009:20:46:56 -0300] “POST /wp-admin/admin-ajax.php HTTP/1.1″ 200 241 “http://www.andresmontalban.com/wp-admin/post.php?action=edit&post=125&message=6″ “Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.2) Gecko/20090803 Linux Mint/7 (Gloria) Firefox/3.0.6, Ant.com Toolbar 1.2″
96.30.25.56 – - [14/Sep/2009:20:47:34 -0300] “GET /wp-content/themes/index.php HTTP/1.1″ 200 – “-” “PHP/5.2.9″
66.249.71.72 – - [14/Sep/2009:20:47:34 -0300] “GET /web-crawlers-speed/ HTTP/1.1″ 200 27611 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

Please let me know your thoughts about this interesting subject, at least for me

Cheers!