GPTBot OAI-SearchBot robots txt: what each bot controls

OpenAI runs four separate web agents, and the rules for each are independent of the others. The trouble with most GPTBot OAI-SearchBot robots txt configurations is that they were written in a single afternoon, when the only question anyone was asking was whether to let an AI company train on their content. The answer was usually no, the block went in, and nobody checked what else it did.

What it often did was remove the site from ChatGPT's search answers. That is a different decision from the training one, it is made by a different crawler, and OpenAI's documentation is unambiguous about the difference.

GPTBot OAI-SearchBot robots txt rules, one bot at a time

OpenAI's crawler documentation lists four user agents and says plainly that each setting is independent of the others. A webmaster can allow one in order to appear in search results while disallowing another to keep content out of model training.

• OAI-SearchBot — search. It surfaces websites in ChatGPT's search features. The documentation states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. If you want to be cited in an answer, this is the one that has to be allowed.
• GPTBot — training. It crawls content that may be used in training OpenAI's foundation models. Disallowing it signals that your content should not be used that way. This is the block most sites actually intended to add.
• OAI-AdsBot — ad safety. It visits pages submitted as ads on ChatGPT to check they comply with policy, and helps decide when an ad is relevant. It visits only pages submitted as ads, and what it collects is not used for model training.
• ChatGPT-User — user-initiated fetches. When someone asks ChatGPT a question and it visits a page in response, this is the agent. OpenAI says it is not used to crawl the web automatically and that, because the action is initiated by a user, robots.txt rules may not apply to it.

That last line deserves a second read. A robots.txt directive is the wrong instrument for stopping ChatGPT from reading a page on a user's behalf — and, separately, ChatGPT-User is not what decides whether content appears in search. OpenAI directs you to OAI-SearchBot for that.

The rule that turns a training block into a search block

The failure is rarely a typo. It is a consequence of how robots.txt group matching works, which most people have never had to think about.

A compliant crawler picks one group: the most specific one matching its own name. It obeys that group and ignores everything else, including the wildcard. So a file that reads User-agent: * with Disallow: /, followed by a friendly User-agent: Googlebot group with Allow: /, is not permissive with one exception. It is a total block with one exception, and every crawler you did not name by hand — OAI-SearchBot included — falls into the blocked group. Google's specification describes the same precedence behaviour.

The mirror-image mistake is just as common. Sites that added a named User-agent: GPTBot block with Disallow: / did exactly what they meant to. Sites that added a broad AI-crawler block, either in robots.txt or at the CDN edge, usually did not — and the edge case matters, because a rule enforced by a WAF never appears in the robots.txt anyone later inspects.

Worth knowing before you test anything: OpenAI says it can take roughly twenty-four hours from a robots.txt update for its systems to adjust for search results. Change the file, then look again tomorrow.

What we found on our own site

We ran the check on this site before publishing, which is the only honest way to write this. Our robots.txt explicitly names fourteen crawlers and allows them: GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Googlebot and the rest. OAI-SearchBot is not among them.

It is harmless here, because our wildcard group allows everything and the file blocks only the admin and review paths. The named groups are decoration. But swap that wildcard for a Disallow: / — which is what a security-minded person adds when asked to lock down crawling — and the list that looks exhaustive turns out to be missing the single entry that governs whether we appear in a ChatGPT answer. We are adding it. The general lesson is that an explicit allowlist is only as good as its last update, and the bot that matters changed after most of these files were written.

One more thing to settle before you edit anything: whether blocking training is worth what it costs you. For a publisher whose archive is the product, the argument for blocking GPTBot is obvious. For a services business whose site exists to be found and quoted, the calculation is different, and "no" is often being chosen by reflex rather than on its merits. Whatever you decide, decide it deliberately and write down why, because the person who inherits the file in two years will otherwise assume it was considered.

The second failure: content the fetcher cannot see

Allowing a crawler is necessary, not sufficient. It also has to find words on the page.

Googlebot renders JavaScript, and has for years, which is why single-page applications rank at all. OpenAI's crawler documentation makes no such commitment — it describes user agents, published IP ranges and robots.txt handling, and says nothing about executing scripts. Absent a statement either way, the safe assumption for anything you want quoted is that the copy needs to be in the HTML the server sends.

This is straightforward to check without any tooling. Fetch your own page with curl, strip the tags, and read what is left. If your value proposition, your prices and your answers to common questions survive that, you are fine. If what remains is a nav bar, a footer and an empty div, then a fetcher that does not run scripts sees a site with nothing to say, whatever your rankings look like. Server-rendering the substantive copy is the fix, and on a modern framework it is usually a configuration decision rather than a rewrite — the same reasoning we apply on any web application build.

Verify the traffic before you trust your logs

Once the rules are right, the follow-up question is whether the bots are actually arriving. User-agent strings are trivially forged, so counting log lines that say OAI-SearchBot tells you very little on its own.

OpenAI publishes the IP ranges for each agent as JSON — searchbot.json, with equivalents for GPTBot, AdsBot and ChatGPT-User. Matching the source address against the published range is the check that means something, and it is the same discipline as verifying Googlebot by reverse DNS. OpenAI also notes that when it fetches robots.txt it may add a marker to the user-agent string, so site owners can tell a robots.txt request apart from a page request even when their logs do not include paths. If you are not logging user agent and source IP for bot traffic, start there, because every other question in this article is unanswerable without it.

A half-hour checklist

1. Open /robots.txt on your production domain and read it as a crawler would: find the most specific group that matches OAI-SearchBot, and check whether that group allows or denies.
2. Check your CDN or WAF for a managed rule that blocks AI crawlers. It will not be visible in robots.txt and it overrides everything.
3. Decide the two questions separately. Training use and search visibility are different trades, and for most businesses the sensible answer is to block training if you want to and keep search open.
4. Fetch a key page with scripts disabled and confirm the copy is there.
5. Log user agent and source IP, and verify against OpenAI's published ranges before drawing any conclusion from the counts.
6. Wait a day after any robots.txt change before re-testing.

None of this is a growth strategy. It is a check that the door is unlocked, and it belongs with the other quiet configuration decisions that nobody owns until they cause a problem — the same category as the governance gaps that build up on a neglected site. The distinction worth keeping is that appearing in an AI answer and being used as training data are two different permissions, controlled by two different agents, and treating them as one setting means you are certainly getting one of them wrong.

If you want a second pair of eyes on how your site reads to a fetcher, or on where the substantive copy actually lives in your markup, that is part of how we think about AI-ready architecture generally — and we are happy to take a look.