Can AI actually read your site? What robots.txt and llms.txt control

Every diagnosis of a missing AI citation starts by assuming the content is the problem: wrong angle, wrong depth, wrong format. Worth checking first is something more boring and more common — whether the crawler was ever let in.

A grounded engine — Perplexity's sonar models, Gemini with search grounding — cannot cite a page it cannot fetch. If your robots.txt blocks the crawler, no amount of content work changes the outcome, because the outcome was decided before the model ever reasoned about your content.

The block nobody remembers adding

Most robots.txt files predate GPTBot, ClaudeBot and PerplexityBot entirely. A rule written years ago to keep out aggressive scrapers, or copied wholesale from another site's config, can end up disallowing exactly the traffic you would want today. Nobody sits down and decides to block an AI crawler; it happens as a side effect of a decision made for an unrelated reason, on a file almost nobody reads after launch.

What to actually check

Fetch your own robots.txt and read it the way a crawler would: as a sequence of User-agent blocks, each with Disallow/Allow rules that apply only to that block, falling back to the wildcard (*) block when a crawler isn't named. Five names are worth checking specifically.

• GPTBot and OAI-SearchBot — OpenAI's crawlers, behind ChatGPT and its search features.
• ClaudeBot — Anthropic's crawler, behind Claude.
• PerplexityBot — Perplexity's own crawler, distinct from its answer-time fetches.
• Google-Extended — governs Gemini and AI Overviews specifically, separate from classic Googlebot indexing.
• Bytespider — ByteDance's crawler, behind Doubao and increasingly relevant outside China.

A rule that disallows one of these by name overrides the wildcard block for that crawler alone — so a site can look wide open in its general User-agent: * rules while specifically excluding the one crawler that matters to you. Reading only the wildcard block is the most common way this gets missed.

This is a site-wide decision, not a page-level one

A single robots.txt at your domain root usually governs the whole site, which is both the convenience and the risk. Fix the one rule and every page benefits immediately — but a single overbroad rule can also silently cost you every page at once, which is a larger blast radius than most teams assume when they add a rule meant for one section.

If you do have per-path rules for a specific crawler, read the file directly rather than trusting a summary — a tool that reports your domain root's rules, including CiteTrail's own free checker, is telling you about the root, not necessarily every path underneath it.

Should you block AI crawlers?

That is a real decision some brands make deliberately, and it deserves an honest answer rather than a reflexive one. Concerns about training-data use, competitive sensitivity, or simply not wanting a particular crawler's traffic are legitimate reasons a site might choose to block some or all of these agents.

What is worth being precise about: blocking a crawler and being invisible to grounded citation are the same lever. If a buyer question gets answered by an engine that reads live documents, and yours is the one document that engine cannot fetch, you have opted out of that specific citation — not adjusted your ranking within it, opted out entirely. Make that trade knowingly, not as an accident of an old scraper rule.

What llms.txt is, and what it isn't

llms.txt is an emerging convention — a plain-text file at your domain root that summarises your site specifically for AI systems, the way robots.txt communicates crawl rules and sitemap.xml communicates page structure. It is not a standard any engine is confirmed to consult as part of answering a query, and treating it as guaranteed AI SEO overstates what is currently known.

The order that actually matters

1. 1Confirm the crawlers you care about can fetch your site at all — a one-minute check against your own robots.txt, or run it through a free checker.
2. 2Confirm the content that answers your highest-value buyer questions is on a page those crawlers can reach, not gated behind a login or a JavaScript-only render they may not execute.
3. 3Only then invest in the content and structured-data work that makes a fetched page easy to cite — see FAQ schema and AI citations for the next layer.

Doing this in the other order — polishing content that a blocked crawler will never see — is the single most avoidable waste of effort in this category of work, and the reason it's worth being the first thing checked, not the last.

None of this proves an engine will cite you once it can reach you; access is a precondition, not a guarantee. What it does prove is whether you have a chance at all — and that is checkable in under a minute with CiteTrail's free AI crawler access checker. For the full picture beyond crawler access — schema, structure, and citability together — the free readiness audit and the scoring behind it on /methodology cover the rest.