Since I setup Google Sitemap and robots.txt across my sites the GoogleBot has become slightly less intense, and is now only responsible for about 25% of the bandwidth that the sites all use! I still think that the spidering is a bit intense for personal web sites, and it seems not to be taking much notice of the sitemaps recommendations, but it is now down into the realms of acceptable behaviour.
tags: blog home computing web
Post new comment