AI crawler access.

Being found starts with being fetched

Everything else in this series is about what happens once an engine reaches your pages — whether the content is structured to be lifted, whether there's outside evidence to corroborate you, whether the words on the page actually answer a real question. None of it matters if the crawler never gets there. That's the layer underneath all the others, and it's the one most business owners assume is already settled and never check again.

It goes unchecked because the rule that's blocking it was usually a good decision at the time. Somewhere in a site's history — often years back, often after a scraping incident or a spike in junk bot traffic — someone added a rule to keep bots out: a wildcard Disallow, applied once, forgotten since. What that decision couldn't have anticipated is that "bot" now includes the crawler that builds the answer a buyer sees when they ask an AI assistant who to hire.

The fix isn't reflexively opening everything back up — some of what that old rule stops is still worth stopping. It's knowing which bot is which before deciding.

Ten crawlers, four companies

Google, OpenAI, Anthropic, and Perplexity don't run one crawler apiece. Each runs several, with distinct jobs, each independently addressable in robots.txt. Getting this wrong — blocking too much, or assuming a block worked when it didn't touch the crawler that mattered — is the single most common access problem a readiness check turns up.

Crawler	Runs for	Job
Googlebot	Google	Crawls and indexes pages for Search
Google-Extended	Google	Governs Gemini's training & grounding use of your content
GPTBot	OpenAI	Collects training data
OAI-SearchBot	OpenAI	Surfaces pages in ChatGPT's search features
ChatGPT-User	OpenAI	Fetches a page a live user's action triggers
ClaudeBot	Anthropic	Training crawler
Claude-SearchBot	Anthropic	Crawls to improve Claude's search results & citations
Claude-User	Anthropic	Fetches a page a Claude user directs it to
PerplexityBot	Perplexity	Builds the index behind Perplexity's cited answers
Perplexity-User	Perplexity	Fetches a page on a live user's request

Googlebot is the one every site already half-knows about — block it and you lose Search indexing outright, which cascades into AI Overviews and AI Mode eligibility, since Google's own documentation states there's no separate technical bar for those beyond ordinary indexing. Google-Extended is a different lever entirely: it controls whether Gemini can train on and ground answers in your content, and Google states explicitly that toggling it does not affect Search ranking or inclusion. You can turn one off without touching the other.

OpenAI splits the job three ways. GPTBot only collects training data — blocking it opts you out of training and does nothing to whether ChatGPT can surface or cite you in a live answer. That's OAI-SearchBot's job, and it's the one that actually gates citation eligibility in ChatGPT's search features. ChatGPT-User is different again: it fires when an actual person's action inside ChatGPT triggers a fetch, so it behaves closer to a browser request than a scheduled crawl.

Anthropic runs the same three-way split. ClaudeBot trains. Claude-SearchBot crawls specifically to improve the quality of Claude's search results and citations. Claude-User fetches a page because a Claude user pointed it at something specific, live.

Perplexity's two bots divide the same way, with one wrinkle. PerplexityBot builds the index Perplexity's cited answers draw from — block it and you're not just declining training, you're removing yourself from discoverability in cited answers entirely. Perplexity-User is documented as generally disregarding robots.txt altogether, on the reasoning that it's fulfilling one person's specific request rather than crawling — worth knowing, because it means a block here isn't the control point some site owners assume it is.

Training and citation are not the same decision

The mistake this catches most often: a site blocks GPTBot, or ClaudeBot, to keep its content out of AI training, and the owner walks away assuming they've also opted out of appearing in that assistant's answers. They haven't. Training crawlers and search-and-citation crawlers are separate bots doing separate jobs, and blocking one has no effect on the other. A business can decline AI training entirely and remain fully visible in AI search and citation — but only by controlling the bots individually. One blanket rule can't make that distinction. It either lets everything through, or blocks the exact crawler that would have cited you.

robots.txt is a request, not a lock

It's worth being precise about what a robots.txt rule actually does, because it does less than it looks like. It's a request that cooperative crawlers agree to honor — not a technical barrier anything is forced through. Most of the crawlers above comply with it. Some don't: Perplexity-User, as noted, is documented as generally ignoring it, since it's serving a live user's request rather than crawling.

That cuts two ways that are easy to conflate. A carelessly written robots.txt can absolutely and accidentally exclude a legitimate citation crawler — that's the common failure, and it costs you visibility with no upside. But robots.txt cannot stop a genuinely non-compliant scraper; that needs network- or firewall-level enforcement, a different tool for a different problem. Don't reach for robots.txt to solve a security problem, and don't assume a security tool is visible in robots.txt at all — a clean robots.txt file means nothing if a CDN's bot-management rule is silently challenging or blocking crawlers before they ever see it. Check both layers, separately.

Snippet directives can quietly cost you the same thing

A page can be fully crawlable and still be excluded from AI features by a second, easy-to-miss layer: directives that control how much of the page is allowed to be shown or quoted. noindex, nosnippet, data-nosnippet, and max-snippet all restrict preview or excerpt rights — and unlike a robots.txt block, these are often not a deliberate decision at all. They're frequently an SEO plugin's default, applied sitewide the day it was installed and never revisited. A page that's perfectly reachable can still be ineligible for the answer-engine feature that would have cited it, because something on the page is telling the crawler it isn't allowed to quote the passage that would have been the citation.

What this looks like in practice

A crawler-access review runs faster than it sounds, and it's worth doing as its own pass rather than folding it into a general SEO check:

• Confirm the site itself is reachable first. Healthy status codes, no login wall in front of key pages, and core content present in the HTML the server actually sends — not only after client-side JavaScript runs. Nothing downstream matters if this fails.
• Read the live, production robots.txt line by line against the crawler list above — the one actually serving, not a staging copy — and flag any blanket rule or any search-oriented crawler blocked intentionally or not. Then check CDN and firewall bot-management settings separately; they sit outside robots.txt entirely, and a clean file there proves nothing about them.
• Check snippet and preview directives on your highest-value pages for anything overly restrictive, especially settings inherited from a plugin default rather than a decision anyone actually made.
• Write the policy down as three separate calls, not one: what should be findable, what should be trainable, what should be protected. For most businesses that want visibility and aren't strongly opposed to training use, that means permitting the search-oriented crawlers and Googlebot, deciding the training question deliberately with the training-specific tokens, and reserving hard blocks — enforced at the network level, not just robots.txt — for content that's genuinely sensitive. Revisit it periodically; provider policies change, and a robots.txt reviewed once and never again drifts the way anything unmaintained does.

Frequently asked

Does blocking an AI training crawler also block me from AI search results?

No — this is the single most common misunderstanding here. GPTBot and ClaudeBot handle training and can be blocked without affecting anything else. Whether you're surfaced in a live search or citation is controlled by a separate crawler entirely — OAI-SearchBot, Claude-SearchBot, or PerplexityBot. Block the wrong one and you'll have accomplished nothing you intended.

Is robots.txt enough to stop a bot I don't want on my site?

For a cooperative crawler, yes — that's exactly what it's for. For a bad actor deliberately ignoring it, no; robots.txt is an honor system, and an actual block requires network- or firewall-level controls. Treat these as two separate problems with two separate fixes, not one setting that solves both.

Does Google-Extended affect my Search ranking?

Google states explicitly that it doesn't. Google-Extended controls Gemini's use of your content for training and grounding; Googlebot is the separate crawler responsible for Search indexing and ranking. Blocking the former leaves the latter untouched.