Free tool
AI crawler checker
Enter a site or a page. We read its robots.txt and tell you which of 18 AI crawlers may fetch it, and which rule decides each one.
See which assistants actually read your pages and what they send you
Pathrail logs every AI crawler visit from your server, verifies it, and shows the visitors and revenue each assistant sends back.
How the check works
We fetch /robots.txt from the site and apply the same rules Google documents: the most specific user-agent group wins, then the longest matching path rule, and Allow beats Disallow on a tie. Wildcards (*) and end anchors ($) are supported.
The 18 crawlers are the ones that matter for AI visibility: OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User), Perplexity, Google-Extended, Applebot-Extended, Meta, Amazon, ByteDance, Common Crawl, Cohere, DuckDuckGo and Mistral.
A crawler being allowed doesn't mean it visits. To see the real fetches, page by page, connect your server logs to Pathrail's AI crawler collector.