Tools / AI Crawler Logs
AI Crawler Log Analyzer
Paste your access log and see how much of the traffic comes from AI crawlers like GPTBot and ClaudeBot, which pages they actually read, and how often they only fetch sitemap.xml and robots.txt.
Everything is parsed by JavaScript in your browser — the log never leaves your device. nginx / Apache combined format and Cloudflare Logpush CSV are supported.
Crawler breakdown
Top crawled paths
Status codes
AI crawler requests per day
Why analyze AI crawler traffic?
Requests from AI crawlers have grown quickly, and they behave differently from Googlebot: site owners publishing their logs report that the majority of AI bot requests go to sitemap.xml and robots.txt while content pages are barely touched. This tool turns a raw access log into that picture — in your browser.
How to use this tool
- Upload or paste an access log: nginx / Apache combined format, or a Cloudflare Logpush CSV export.
- Pick the log format, or leave auto detect on.
- Read the summary — AI crawler share, the busiest bot, and how much of their budget goes to sitemap.xml and robots.txt.
- Export the crawler table as CSV, then decide whether to allow, throttle or block each bot in robots.txt.
FAQ
Is my log file uploaded anywhere?
No. The file is read with the browser File API and parsed by JavaScript on your device, so nothing is transmitted. That also means it works offline and on a private staging site.
Which log formats are supported?
The nginx and Apache combined format — the default for most hosts, including /var/log/nginx/access.log — and CSV exports such as Cloudflare Logpush with ClientIP, ClientRequestPath, ClientRequestUserAgent and EdgeResponseStatus columns.
Does blocking these bots hurt my SEO?
It depends on the bot. GPTBot, ClaudeBot and CCBot collect training data, and blocking them does not change classic search rankings. OAI-SearchBot, PerplexityBot and Googlebot feed search surfaces, so blocking those can remove you from AI answers. Test the rule first in the robots.txt generator.