How answer engines cite: the mechanism, the controls, the guesswork

What this is about. Answer engines do not run a second internet. They read the same pages your readers read, through the 10 named crawler tokens listed below, and then quote a narrow slice of what they read. This page sets out the mechanism, the controls that exist, and the parts nobody can honestly promise you.

On this page

• How does an answer engine actually work?
• The crawlers, and which control does what
• What earns a citation?
• What can you opt out of, and what can you not?
• The standards that are not standards yet

How does an answer engine actually work?

Strip the branding away and every answer engine runs the same three steps. It retrieves candidate documents for the question. It grounds a generated answer in those documents. Then it cites some of them, usually fewer than it used.

The retrieval step is the one SEO already knows. Google’s AI Overviews retrieve from the Google Search index, so the pages that can appear are pages Googlebot has crawled and indexed. Perplexity and ChatGPT search run their own retrieval over their own crawls, plus licensed search partners. Different index, same dependency: something has to have fetched your page and kept it. Retrieval is also more concentrated than it looks: across a 145-million-result analysis, three domains supplied 76.75% of every link in Google’s Discussions and Forums feature.

The grounding step is where the shape of your page starts to matter. A model composing an answer needs a passage that states the thing plainly. A page that buries its answer in the eleventh paragraph is harder to ground against than a page that answers in 40 to 60 words and then explains itself. Maintenance matters as much as wording: across 7,683 dated pages carrying 47,097 citations, 75% of the pages engines cited had been updated within the past year.

The citation step is the one nobody outside the vendors can see, and sometimes there is no citation at all: across 1,562 branded prompts, engines declined to answer 31.2% of the time and answered wrongly only 2.8%. Two engines given the same retrieved set cite differently, and the same engine cites differently on the same question a day later. Treat published claims about citation weighting with the suspicion you would give any unaudited vendor number, and be equally careful about what can and cannot be measured downstream. One site owner tracked a 1,900% month-over-month rise in ChatGPT citations to one page and measured almost no business effect.

The crawlers, and which control does what

OpenAI documented GPTBot in August 2023 and Google published Google-Extended in September 2023, and the list has grown every year since. The single most common mistake in this area is treating one token as if it did two jobs. Fetching a page, using it to answer a live question, and using it to train a model are three different things, and vendors expose different tokens for them. Blocking the wrong one either does nothing or removes you from the results you wanted.

Token in robots.txt	Owner	What it governs
Googlebot	Google	Search crawling. Also the source of AI Overviews eligibility.
Google-Extended	Google	Training and grounding for Gemini. Not a crawler; does not affect Search or AI Overviews.
GPTBot	OpenAI	Crawling for model training.
OAI-SearchBot	OpenAI	Crawling for the ChatGPT search index.
ChatGPT-User	OpenAI	Fetches made because a user asked, in the moment.
ClaudeBot	Anthropic	Crawling for model training.
Claude-User	Anthropic	Fetches made on a user’s request.
PerplexityBot	Perplexity	Crawling for its search index.
Applebot / Applebot-Extended	Apple	Crawling, and separately training use.
CCBot	Common Crawl	An open corpus many models are trained on.

Read the third column twice before editing a file. For how the rest of the web actually sets these rules, see what the 2025 Web Almanac found about bot access decisions. Blocking GPTBot keeps your text out of training runs and leaves ChatGPT search able to cite you, because that is OAI-SearchBot‘s job. Blocking Google-Extended makes no difference to whether you appear in an AI Overview. The two decisions people usually want to make — “do not train on me” and “do cite me” — are compatible, and the tokens are how you say both.

A robots.txt rule is a statement of policy, not a lock. It is obeyed by the crawlers that choose to obey it.

What earns a citation?

Here is the honest division. Some things are documented and controllable. Some are observable but unexplained. The rest is marketing.

Documented and controllable

• Being crawlable and indexable at all. Every citation route starts here, and it is where most sites actually fail.
• Not suppressing your own snippets. nosnippet, a restrictive max-snippet, or data-nosnippet around the substance will remove the passage an engine would have quoted.
• Rendering without requiring script execution for the main text. A page whose body arrives only after JavaScript runs is a gamble on each crawler’s rendering budget.
• Stating the answer in prose near the top. This is not a trick; it is what makes a passage quotable.

Observable but unexplained

The one controlled test we have found points the other way on markup: Ahrefs compared 1,885 pages that added JSON-LD with 4,000 matched controls and measured a 4.6% fall in AI Overview citations, the only statistically significant result. Practitioners report that engines lean on sources that state numbers with a method attached, and on pages narrow enough to be about one question. That matches how retrieval and grounding work, and it is a reasonable way to write regardless. It is not a documented ranking factor, and this site will not present it as one.

Marketing

Any product promising to place you in AI answers, any percentage attached to a “citation score”, any claim that a schema type causes citation. No vendor exposes the mechanism, so no tool can target it.

What can you opt out of, and what can you not?

You can refuse training use, per vendor, with the tokens above. You can refuse crawling entirely, at the cost of the citations that crawling enables. What you cannot currently do on Google is stay in ordinary search results while excluding yourself from AI Overviews: the controls that suppress the AI snippet are the same controls that suppress the search snippet.

That is a real trade-off and it deserves a decision rather than a default. For a publisher whose traffic depends on the click, it is worth modelling before touching anything. For most sites the calculation is simpler: appearing is better than not appearing.

The standards that are not standards yet

llms.txt is a proposed convention: a plain-text map of your site at the root, written for models rather than browsers. No major engine has committed to reading it, which puts it alongside the other GEO claims that thin out under testing, and one analysis of 137,000 sites found 97% of published llms.txt files are never fetched. It is cheap, it does no harm, and it is not a ranking factor. Publish it because it forces you to state what your site is about in one page, not because it will move anything.

The same caution applies to every “AEO checklist” that appeared in the last year. The controls that exist are the ones vendors document. Everything else is a hypothesis, and hypotheses belong in a test, not in a client report.

FAQ