Can AI crawlers actually read your site?
An AI assistant can only cite what it can read. Check whether your robots.txt is quietly blocking GPTBot, ClaudeBot, PerplexityBot and the rest — and whether you're publishing an llms.txt.
We read your site's public robots.txt and llms.txt server-side — the same files any crawler would check first.
A block nobody remembers adding.
Most robots.txt files predate these crawlers entirely. A rule written years ago to keep out scrapers can end up blocking the exact traffic you'd want today — and nothing about a blocked crawler looks like an error, it just quietly never reads your pages.
Found a block?
Removing a Disallow rule for a specific crawler is usually a one-line change in robots.txt. If you're not sure which line is doing it, the file itself will show a User-agent block naming that crawler.
Want the full picture, not just robots.txt?
Crawler access is one input into whether AI assistants recommend you at all. CiteTrail's free readiness audit scores your site across schema, structure, and citability — and checks the same crawler rules alongside everything else.
Run the free readiness auditNeed paste-ready structured data to help engines cite you too?
Try the free FAQ schema generatorAI crawler access, answered honestly.
If something here doesn't match what the tool does, that's a bug — tell us.
What is this checking, exactly?
It fetches your site's public robots.txt and looks for rules that block the AI crawlers behind ChatGPT, Claude, Perplexity, Google's AI features, and Doubao — the same file any of those crawlers reads first. It also checks whether you publish an llms.txt.
Why would my robots.txt block an AI crawler?
Usually a boilerplate 'block everything unrecognised' rule, added before these crawlers existed, or a blanket disallow copied from another site's config. It is rarely a deliberate decision — which is exactly why it is worth checking.
Should I block AI crawlers?
That is a real decision, not a default — some brands choose to opt out of some or all of them. This tool tells you the current state, not what it should be. If you want AI answers to be able to cite you, GPTBot, ClaudeBot and PerplexityBot being blocked is usually working against that goal.
What is llms.txt, and do I need one?
It is an emerging convention — a plain-text file at your domain root that summarises your site for AI systems, the way robots.txt does for crawlers. It is optional, not a standard any engine is confirmed to consult yet, but it costs little to publish and is the one file written specifically with AI in mind.
Does this tool check every page on my site?
No — it reads robots.txt and llms.txt at your domain root. A single robots.txt rule usually governs a whole site, but if you have per-path rules for these crawlers, check the file directly for anything this summary would miss.