botstat

seo bot crawl rate optimization

There was a time when we needed to detect Google’s crawl rate faster than webmaster tools allowed to check. I was telling about the small re-incarnation of the BotStat idea using Python. The initial post was here. Right now it works for Nginx and Apache with detecting your access logs formats automatically (could…