AI crawler

Three jobs, three kinds of bot

Most of the contradictory advice about AI bots comes from people describing different jobs and using the same word for all of them. The useful way to sort them is by what the fetch is *for*, because that is what decides the cost of turning it away.

• Training crawlers collect text that may end up in the training data of a model that does not exist yet. OpenAI's GPTBot, Anthropic's ClaudeBot and the Common Crawl project's CCBot are the ones named most often. Turning them away changes nothing about whether you are quoted today.
• Index crawlers build the retrieval index an answer engine searches at the moment it answers — OpenAI's OAI-SearchBot and PerplexityBot are the clearest examples. This is the group whose absence costs you AI citations, and it is the group people block by accident.
• User-triggered fetchers pull a single page because a person pasted a link or asked a question about it, which is why OpenAI names its one ChatGPT-User. Functionally these are closer to somebody clicking a link than to a crawl, and they arrive one page at a time.
• The ones that never say. A large share of automated traffic presents itself as an ordinary browser, or sends no identification at all. No robots rule reaches them, because a robots rule is a request addressed to a name.

What robots.txt actually decides, and what it does not

robots.txt is a set of per-user-agent requests, honoured voluntarily by whoever chooses to read it. It governs fetching, and only fetching. It cannot reach back and remove text already collected, it cannot bind a client that ignores it, and — worth remembering before you write a long one — it is a public file, so a list of the paths you would rather nobody read is itself readable by anybody.

The control founders most often misread is Google-Extended, because it is not a crawler at all. Google documents it as a token that governs whether content Googlebot already fetched may help improve its generative products, and states that it does not affect inclusion in Search. It is therefore not the switch for Google's AI Overviews, which are generated from the ordinary Search index and governed by the ordinary snippet controls. Adding Google-Extended does less than most people think, and reaching for Googlebot instead to get the effect they wanted would remove them from Search altogether.

The same asymmetry runs through the whole file. Every rule you write is aimed at a bot that identifies itself honestly and chooses to obey — which is to say, at the well-behaved end of the traffic, and not at the end most likely to be the reason you started reading about this.

So should you block them?

What makes this decision awkward is that it is asymmetric in time. Allowing a crawler has a visible, immediate cost — requests hitting your server this afternoon. Blocking one has an invisible cost that arrives months later, as answers that quote somebody else. Three cases separate cleanly:

• You sell to people who ask assistants questions. Allow the index crawlers, and stop worrying about the training ones. Being retrievable is the precondition for everything else in generative engine optimization; there is no version of being cited that starts with not being read.
• Your content is the product — a paid archive, a course, a dataset you license. Now the training question is a real commercial question rather than a reflex, and declining is defensible. Be clear-eyed that it is forward-looking only, and that the index crawlers are a separate decision you can still answer differently.
• Your actual worry is server load. Then the named AI bots are probably not your problem, and the diagnosis belongs with crawl budget and your origin logs rather than with a robots rule.

For most small startups the honest default is to allow the named crawlers and spend the saved afternoon on being worth quoting — a clean, factual page that answers one question in one liftable passage does far more for citation than any robots rule can. A llms.txt file is a cheap addition on top, not a substitute for being fetchable.

What we learned: the list decides, and the list can be wrong

AgentCeres — the AI Growth Officer at agentceres.com — ran into the sharp edge of this on its own marketing site. Our framework serves slightly different HTML to crawlers than to browsers, decided by a built-in list of crawler user agents. Googlebot was not on that list by default, while the agent behind Search Console's live-URL test was. So the live test rendered a flawless page every time we checked, and real crawls received the canonical and hreflang tags below the closing head tag, where they do not count. Ninety-odd pages were logged as duplicates without a user-selected canonical before we found it, and the reason it survived so long is that the tool we verified with sat on a different list than the crawler we cared about.

The generalisation is worth more than the bug. Any behaviour that treats one crawler differently from another depends on a name matching a list, and both halves rot: bots get renamed and split, and lists are written by people guessing at next year's names. There is a second trap underneath it — our CDN caches by URL and not by user agent, so whichever variant of a page is cached first is what every later visitor and every later bot receives. Per-crawler behaviour is considerably easier to configure than it is to actually deliver, which is one more argument for a site that serves everybody the same honest page.