← Back to blog

Technical SEO · August 10, 2026 · 7 min read

llms.txt Explained: Should You Add One in 2026 (and What to Put in It)?

Learn what llms.txt does, which AI crawlers support it, and get a tested file template with pitfalls to avoid in 2026.

By FluxWriter Team

The llms.txt file is a proposed web standard that lets site owners declare what AI language models should — or shouldn't — do with their content. It's modeled loosely on robots.txt, but designed for a world where the crawlers consuming your pages aren't returning search-result snippets; they're training datasets or real-time retrieval systems. Before you rush to create one, it's worth separating what llms.txt actually does today from what its proponents hope it will do.

What Is llms.txt, Exactly?

The llms.txt proposal was introduced by Jeremy Howard in September 2024. The specification calls for a plain-text (optionally Markdown) file placed at the root of a domain — https://yourdomain.com/llms.txt — that describes the site and links to key content in a format optimized for LLM ingestion rather than human browsing.

The canonical structure looks like this:

# Site Name

> One-sentence description of the site.

## Docs
- [Getting Started](https://example.com/docs/start): Core onboarding guide.
- [API Reference](https://example.com/docs/api): Full parameter reference.

## Optional
- [Changelog](https://example.com/changelog): Version history.

Sections marked ## Optional can be skipped by a token-limited AI agent. That's the primary differentiator from a sitemap: llms.txt is curated and context-rich, not exhaustive.

There is also a sibling convention, llms-full.txt, which embeds the full content of each linked document inline — useful for AI systems that can ingest a large context window in a single request rather than crawling page by page.

Who Supports It Right Now?

This is where things get nuanced. As of mid-2026, adoption looks like this:

The honest picture: llms.txt is widely adopted by developers and documentation sites but is not yet a signal that any major AI crawler formally processes as a ranking or crawl-priority input. Anthropic has written about it favorably but has not published a spec for how Claude's systems weight it. There is no RFC, no W3C draft, no IANA registration.

That's not a reason to ignore it — but it is a reason to have accurate expectations.

Does llms.txt Actually Influence AI Crawlers?

Direct evidence is thin. The most concrete data point available is from the llms.txt directory maintained at directory.llmstxt.cloud, which tracked over 15,000 published llms.txt files by Q1 2026 — showing that adoption is real, even if validation from AI vendors lags behind.

What we do know mechanically:

• Agentic AI tools (Claude Projects with web browsing, GPT-4 with Browsing, Perplexity) fetch URLs dynamically. If they land on your homepage and follow a link to llms.txt, they can use it to orient themselves. Whether they do so automatically or only when triggered by user instruction varies by implementation.
• Training data pipelines (Common Crawl-based) crawl at scale and do not currently parse llms.txt as a directive. A User-agent: * disallow in robots.txt is still the only reliably honored signal for opt-out.
• RAG retrieval systems built on your own content (internal enterprise tools, custom GPTs with file uploads) are unaffected — they consume content you explicitly provide.

The practical upshot: llms.txt has more immediate value for agentic retrieval — AI assistants actively exploring your site to answer a user query — than for influencing training data inclusion or SEO rankings.

Why Add One Anyway

Even without confirmed crawler support, there are defensible reasons to publish llms.txt today:

Forward compatibility. If Anthropic, OpenAI, or Perplexity formalizes support in the next 12 months (plausible, given stated interest), early adopters get indexed behavior that reflects their own curation rather than whatever the crawler inferred from page structure.

Developer and API documentation sites. AI coding assistants are already used heavily to navigate technical docs. A well-structured llms.txt gives those assistants a reliable entry point without having to reverse-engineer your navigation. Stripe, Vercel, and Astro all published llms.txt files within weeks of the proposal going public.

Controlled narrative. Your llms.txt is a brief, accurate, context-rich description of your site that you wrote. That's better than an AI system summarizing you from a misread landing page.

Low cost. For most sites, creating a useful llms.txt takes under an hour.

A Tested File Template

Here is a production-ready template with notes on what to include and what to leave out:

# [Site Name]

> [One sentence: what the site does and for whom. Keep under 120 characters.]

## Core Pages
- [Home](https://example.com/): Overview and entry point.
- [About](https://example.com/about): Mission and team background.

## [Primary Content Category, e.g. "Blog"]
- [Most important post title](https://example.com/blog/post-slug): [10-word description].
- [Second most important post](https://example.com/blog/post-slug-2): [10-word description].

## Docs
- [Getting Started](https://example.com/docs): Where new users should begin.
- [API Reference](https://example.com/docs/api): Full method and parameter reference.

## Optional
- [Changelog](https://example.com/changelog): Release notes and version history.
- [Sitemap](https://example.com/sitemap.xml): Full URL index.

What to Omit

• Do not list every post or product page. Curate to the 10–20 URLs that define your site.
• Do not include login-gated URLs — AI agents cannot authenticate.
• Do not add marketing copy to descriptions. Use factual, noun-phrase descriptions.
• Do not mirror your sitemap.xml. The value of llms.txt is editorial selectivity.

Where to Serve It

The file must be at the root path. For a Next.js app, place it in /public/llms.txt. For a WordPress site, use a plugin or add a rewrite rule. Confirm it's accessible with curl -I https://yourdomain.com/llms.txt — you want a 200 OK, not a redirect chain.

Common Pitfalls

Treating it like robots.txt for opt-out. llms.txt has no established enforcement mechanism. If you want to block AI training crawlers, robots.txt with specific User-agent directives (e.g., GPTBot, ClaudeBot, PerplexityBot) is the only option with documented effect.

Publishing a stale file. A llms.txt that links to outdated or removed pages undermines the purpose. Add it to your content publishing checklist; when a key page changes, update the file.

Using it to hide low-quality pages. Some site owners have floated llms.txt as a way to steer AI away from thin content. That content is still crawlable by all other means. Fix the content; don't try to route around it.

Confusing it with the X-Robots-Tag header or noindex directives. Those are search engine signals. llms.txt is a separate, parallel convention with a different (and currently less mature) ecosystem.

FAQ

Does adding llms.txt help my Google rankings? No. Google's crawl pipeline does not read or act on llms.txt. It has no effect on traditional search rankings. Its scope is AI agent and LLM retrieval behavior, not web search indexing.

Should I include llms-full.txt as well? Only if your site has dense technical documentation and you want to support single-fetch ingestion by high-context-window models. For a typical blog or marketing site, the standard llms.txt with curated links is sufficient and much easier to maintain.

Can I block specific AI bots with llms.txt? No. llms.txt is not an access-control mechanism. Use robots.txt with named User-agent strings — GPTBot, ClaudeBot, PerplexityBot, Bytespider, etc. — to block specific crawlers. Most major AI vendors have committed to honoring robots.txt disallows.

Practical Takeaway

Publish an llms.txt file if you run a documentation site, a technical blog, or any site where AI assistants are likely to be used as navigation tools. The file takes an hour to write, costs nothing to serve, and puts your own editorial judgment in front of the systems that increasingly mediate how people find and understand your content. Hold off on llms-full.txt unless you have a genuine use case for single-fetch ingestion.

Don't expect it to move your rankings or block training data — those outcomes require different tools. What llms.txt does well today is give agentic AI systems a reliable, human-curated map of your site, which is a genuinely useful thing to have as that category of tool matures.

If you write a lot of content and want help drafting accurate, factual descriptions for each URL in your file, a tool like FluxWriter can generate those concisely without the marketing padding that tends to creep in when you write them yourself under deadline.

← All posts