Technical SEO · August 11, 2026 · 7 min read
Should You Block AI Crawlers? GPTBot, ClaudeBot and Google-Extended in robots.txt
Learn whether to block AI crawlers like GPTBot, ClaudeBot, and Google-Extended—exact robots.txt directives and a clear cost-benefit guide.
By FluxWriter Team
Deciding whether to block AI crawlers is one of the more consequential choices in technical SEO right now—mostly because you can't easily undo the effects on either side of the equation. Block AI crawlers too aggressively and you lose citation visibility in AI-generated answers; let them through unchecked and your training-data contribution is free, permanent, and uncompensated. This guide walks through the actual trade-offs and gives you the exact robots.txt directives you need.
What These Bots Actually Do
Before writing a single disallow rule, it helps to understand what each bot is collecting and how that collection translates (or doesn't) into traffic or revenue for you.
GPTBot — OpenAI's crawler, used to build and refine GPT model training corpora. Content ingested by GPTBot may end up shaping ChatGPT responses. It does not feed ChatGPT's browsing mode (that's a separate live-fetch mechanism). Blocking GPTBot won't stop ChatGPT from quoting your site from its already-trained weights.
ClaudeBot — Anthropic's crawler, analogous function to GPTBot. Feeds training data for Claude models. Same caveat: past training runs are baked in regardless of what you do today.
Google-Extended — Distinctly different from the others. This is a token that Google lets publishers use to opt out of Gemini and Vertex AI training specifically, without affecting standard Googlebot crawling. Your pages still rank; you're just opting out of AI product training.
CCBot — Common Crawl's bot. Not directly an AI company, but its open dataset is widely used to pre-train large language models. Many AI training pipelines start with Common Crawl snapshots.
The Cost-Benefit Breakdown
What You Lose By Blocking
The most-cited fear is losing mentions in AI-generated answers. That fear is partially valid but often overstated.
AI systems like ChatGPT and Claude are primarily trained on historical snapshots, not a rolling window of current crawls. Blocking GPTBot today doesn't scrub your content from models trained in 2023 or 2024. The practical effect of blocking shows up in future training runs—meaning the impact timeline is months to years, not immediate.
What you do lose more concretely with some bots:
- Google-Extended: If you block this, Google states explicitly that Gemini and AI Overviews products may use your content less for grounding responses. That has measurable implications for publishers who want their brand or content cited in Google's AI features.
- Bing-based AI (Bingbot): Blocking standard Bingbot hurts Bing rankings and therefore Copilot grounding, since Copilot uses Bing's index for retrieval.
What You Gain By Blocking
The direct gain is refusal of free training data. If your content is your primary business asset—proprietary research, paid content, unique datasets—allowing AI companies to train on it without compensation or attribution is a real business cost, not a hypothetical one.
Secondary gain: some bots are bandwidth heavy. GPTBot has been observed crawling at aggressive rates on smaller sites with inadequate crawl management. Blocking reduces server load.
There's also a competitive angle. If you publish in a niche where your content is distinctive, allowing AI training means your differentiation may be blended into a model that competes with you for the same user queries.
Exact robots.txt Directives
Here are the production-ready directives, verified against each company's published documentation.
Block GPTBot (OpenAI)
User-agent: GPTBot
Disallow: /
To allow GPTBot but restrict specific directories (e.g., keep it off paywalled content):
User-agent: GPTBot
Disallow: /members/
Disallow: /premium/
Allow: /
Block ClaudeBot (Anthropic)
User-agent: ClaudeBot
Disallow: /
Anthropic also uses anthropic-ai as a secondary user agent string in some contexts. To be thorough:
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
Block Google-Extended
User-agent: Google-Extended
Disallow: /
Critical note: this has zero effect on regular Googlebot. Your organic search rankings are untouched. This only signals to Google's AI training pipelines. If your goal is to stay out of AI training while maintaining full search visibility, this is the lowest-risk block you can add.
Block CCBot (Common Crawl)
User-agent: CCBot
Disallow: /
Block All Four Together
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
Allowlist Variant (Selective Blocking)
If you want AI crawlers to index your blog but not your product or checkout pages:
User-agent: GPTBot
Disallow: /pricing/
Disallow: /checkout/
Disallow: /account/
Allow: /blog/
Allow: /
A Concrete Example: SaaS Documentation Site
A SaaS company with 400 pages of technical documentation faces a clear decision matrix:
| Scenario | Block GPTBot? | Block Google-Extended? | Outcome |
|---|---|---|---|
| Docs are public, SEO-driven | No | No | Maximum AI citation chance; training contribution free |
| Docs behind login wall | Yes | Yes | Protects proprietary content; minimal SEO impact |
| Docs public, competitive IP | Yes | No | Stops training crawl; keeps AI Overviews eligibility |
| Docs public, brand citation priority | No | No | Prioritize visibility in generative answers |
For most SaaS companies where documentation is a deliberate SEO acquisition channel, blocking is likely counterproductive. For a company whose docs contain hard-won proprietary methodology, blocking is defensible even at the cost of some AI citation.
What robots.txt Can't Do
This is important: robots.txt is voluntary. Reputable AI companies—OpenAI, Anthropic, Google—honor it. Less reputable scrapers, or companies based in jurisdictions with no meaningful IP enforcement, do not.
If you have truly sensitive content, robots.txt is a signal, not a lock. Use authentication, IP rate limiting, and legal Terms of Service language alongside it.
Also worth noting: blocking a bot does not remove content from existing model weights. OpenAI has no mechanism to retroactively remove specific URLs from GPT-4's training data based on current robots.txt state.
Monitoring Crawler Activity
Before deciding, look at your actual logs. In most web server logs or CDN dashboards, you can filter by user agent. Check:
grep -i "GPTBot\|ClaudeBot\|Google-Extended\|CCBot" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn
If GPTBot is making 50,000 requests per month on a 10,000-page site, that's notable. If it's 200 requests, the bandwidth argument evaporates.
FAQ
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended is explicitly scoped to AI training products—Gemini, Vertex AI, and related services. It has no effect on Googlebot's indexing or ranking signals. Google has confirmed this in its crawler documentation.
If I block GPTBot now, will ChatGPT stop mentioning my site?
Not immediately, and possibly not for years. Current ChatGPT responses draw on training data from past crawls. Blocking GPTBot affects future training datasets. The practical impact depends on OpenAI's retraining schedule and whether your content would otherwise be selected for inclusion. ChatGPT's live browsing feature uses a separate real-time fetch mechanism and is not governed by robots.txt in the same way.
Can I block AI crawlers for some pages but not others?
Yes, and this is often the right answer. Use Disallow: directives scoped to specific path prefixes—your /premium/, /members/, or proprietary /research/ directories—while keeping public blog and marketing content open. The partial-block approach gives you protection where it matters without sacrificing discoverability everywhere.
The Practical Takeaway
Block Google-Extended if your concern is AI training but you can't afford any search visibility risk—it's the cleanest, lowest-stakes block available. Block GPTBot and ClaudeBot if your content is a genuine competitive asset and you're willing to accept reduced future training-set inclusion. Leave everything open if AI citation and brand visibility in generative answers is part of your distribution strategy.
There's no universally correct answer. The right call depends on what your content is worth, who's reading it, and what your growth channel priorities are. If you're producing content at scale and want a faster way to test different robots.txt configurations across multiple domains or content types, tools like FluxWriter let you iterate on output quickly so the content strategy decisions stay focused on signal, not production bottlenecks.