llms.txt: What It Is and Do You Need One in 2026?

In short: the llms.txt file is optional. No effect on citation has been demonstrated, and Google states that no special file is needed to appear in its AI features. What really matters is your robots.txt: AI search crawlers must be allowed there.

Before asking whether AI engines understand your content, there's a more fundamental question: can they even access it? A surprising number of well-optimized sites inadvertently block AI bots in their robots.txt file. Others let bots in but serve them unrendered JavaScript — gibberish to crawlers.

This technical guide covers everything you need to know about AI accessibility: the user-agents to know, the correct robots.txt configuration, the llms.txt file (and what to expect from it), and the checks to run to make sure your site is actually indexable by LLMs.

It focuses on the files you control (robots.txt, llms.txt, sitemap). If these files are correct but AI bots are still blocked, the cause lies elsewhere: firewalls, Cloudflare bot protection or JavaScript rendering. These technical causes are covered in why your site blocks AI bots without you knowing.

→Being readable by AI is the prerequisite of the Detekia GEO score — see the full methodology

AI bots: who are they?

Every major AI platform deploys its own bots to crawl the web. Just like Googlebot for SEO, these bots identify themselves with a specific "user-agent" in their HTTP requests.

The distinction that matters is the one between the two columns. Search crawlers read your pages when an AI prepares its answer: if they are blocked, you cannot be cited. Training crawlers collect content to train models: blocking them is a legitimate choice, with no effect on citations. OpenAI spells it out: GPTBot is used for training, and a site that blocks it can still appear in ChatGPT search, which relies on OAI-SearchBot (OpenAI documentation). Likewise, Google-Extended has no effect on inclusion or ranking in Google Search, and therefore none on AI Overviews: it governs the use of your content by Gemini (Google documentation).

The critical point: if your robots.txt contains a Disallow: / directive for User-agent: * (all bots), it blocks all AI bots too. This is often a legacy configuration error — originally meant to block scrapers — that unintentionally applies to LLMs.

Configure robots.txt correctly

A GEO-compatible robots.txt must let AI search crawlers through. Training crawlers are a separate choice: the configuration below allows them, and you can switch their lines to Disallow: / without losing citations.

# robots.txt — GEO-compatible configuration

# Traditional search engines
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# AI search crawlers (required to be cited)
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# Model training (optional: your choice)
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: CCBot
Allow: /

# Google-Extended: use by Gemini, no effect on Google Search
User-agent: Google-Extended
Allow: /

# Default rule
User-agent: *
Allow: /

Sitemap: https://www.your-website.com/sitemap.xml

If you want to allow AI bots while blocking certain scrapers, you can combine specific directives with a restrictive default rule:

# Keep classic search engines
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Explicitly allow AI search crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Block unidentified scrapers
User-agent: *
Disallow: /

Important: Specific directives take priority over general ones. A bot listed explicitly with Allow: / will be allowed even if the User-agent: * rule blocks it. So remember to also list Googlebot, Bingbot and OAI-SearchBot (the ChatGPT search bot), as well as the Claude and Perplexity search crawlers: otherwise a general Disallow: / rule would block your classic search rankings and your citations in ChatGPT.

Check your current configuration

To test your current robots.txt, go directly to https://your-website.com/robots.txt. Look for the user-agents listed above and check whether they're allowed or blocked.

Three situations to identify:

• User-agent not mentioned → the bot inherits the User-agent: * rule. If that rule is Allow: /, you're fine. If it's Disallow: /, the bot is blocked.
• User-agent with Disallow: / → the bot is explicitly blocked. Fix this immediately if it is a search crawler; for a training crawler, it is a valid choice.
• User-agent with Allow: / → correct, the bot can crawl your site.

Google Search Console offers a robots.txt report to check the file read by Googlebot. For non-Google AI bots, you can use the "robots.txt viewer" Chrome extension or a service like sitechecker.pro.

The llms.txt file: a proposal, not a requirement

The llms.txt file is an initiative proposed in September 2024 by Jeremy Howard (co-founder of fast.ai) to create a standard that allows websites to communicate directly with LLMs. It sits at the site root, like robots.txt, and contains a structured summary of the site and its key content.

What llms.txt contains

The proposed format is simple — structured Markdown with defined sections:

# Site Name

> A concise description of what your site does (2-3 sentences max).
> Ideally: who you are, what you offer, who you serve.

## Main Content

- [Complete GEO Guide](https://your-website.com/blog/geo-guide-complete): The definitive guide to optimizing AI visibility in 2026
- [Free GEO Score](https://your-website.com/): Automated GEO score analysis tool, 6 criteria measured
- [Methodology](https://your-website.com/methodology): Detailed explanation of each GEO score criterion

## About

- Founded: 2025
- Expertise: GEO (Generative Engine Optimization), SEO, AI visibility
- Contact: contact@your-website.com

## What We Don't Do

- We don't do paid search advertising (PPC)
- We don't offer content creation services

## Useful Links

- [Blog](https://your-website.com/blog): Technical articles on GEO
- [Pricing](https://your-website.com/pricing): Our plans
- [Contact](https://your-website.com/contact): Contact form

llms.txt vs llms-full.txt

The standard proposes two variants:

• llms.txt — short summary, index of important pages. Ideal for LLMs doing quick scans.
• llms-full.txt — complete version with the full content of key pages. Intended for LLMs that want to index content in depth.

Start with llms.txt. The full version is optional and mainly useful for content-rich sites (documentation, knowledge bases).

Is llms.txt already adopted by AI engines?

As of 2026, llms.txt is still a proposal: many websites and documentation publishers have adopted it, but no major AI engine (OpenAI, Anthropic, Google, Perplexity) has officially confirmed using it to select its sources, and no effect on citation has been demonstrated. Google is explicit: to appear in its AI Overviews and AI Mode, you do not need to create new "AI text files" or add special markup (Google Search Central). All engines, however, respect robots.txt.

The pragmatic recommendation: llms.txt is optional. If you have 20 minutes, creating one won't hurt your SEO, but do not expect citations from it. Your time is better spent elsewhere: AI search crawlers allowed in robots.txt, content readable without JavaScript, then pages that answer clearly under each heading.

Other crawlability obstacles

Client-side JavaScript

This is the most underestimated problem. If your content is rendered in client-side JavaScript (React, Vue, Angular without SSR), basic AI bots won't see that content — they receive the initial HTML without waiting for JS execution.

<!-- ❌ Content invisible to basic bots -->
<div id="app"></div>
<script>
  // Content is loaded after the fact via JS
  document.getElementById('app').innerHTML = '<h1>My content</h1>';
</script>

<!-- ✅ Content visible immediately -->
<h1>My content</h1>

Fix: Use Server-Side Rendering (SSR) or Static Site Generation (SSG). Next.js, Nuxt.js, and Gatsby are built for this. If you're running a pure SPA, implement pre-rendering.

Content behind authentication

Pages that require a login are inaccessible to bots by definition. If your valuable content is behind a login wall, consider making a public version (or a preview) available without authentication.

4xx and 5xx errors

Pages that return HTTP errors will be ignored. Regularly check that your important pages return a 200 using Google Search Console or a crawl tool like Screaming Frog.

Canonicals and redirects

If your important pages have redirect chains or canonical tags pointing to a different URL, bots may not follow through to the final content. Simplify your URL structure and limit redirects to a single hop.

AI accessibility checklist

• ✓ robots.txt tested — OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Googlebot, and Bingbot allowed
• ✓ Key pages without a noindex tag
• ✓ XML sitemap present and referenced in robots.txt
• ✓ (Optional) llms.txt created at the site root
• ✓ Main content rendered server-side (SSR or SSG)
• ✓ No important content behind client-side JavaScript only
• ✓ Key pages return HTTP 200
• ✓ No redirect chains to key pages
• ✓ Response time under 3 seconds

An accessible site is not necessarily a cited site. Once the doors are open, you still need to know whether ChatGPT, Gemini, Claude or Perplexity actually cite you on your key queries: that is the purpose of an AI visibility audit of your brand, which also compares your presence with your competitors'.

To go further on AI access to your site:

• Sitemap and robots.txt: the 2026 configuration for AI bots
• Firewalls, Cloudflare, JavaScript: why your site blocks AI bots
• Website redesign: don't lose your AI visibility (redirects, migrations)
• Internal linking and site structure for AI
• WordPress and GEO: plugins and optimizations
• Schema.org and AI: the practical structured data guide

→Check your site's AI accessibility with the free GEO score

Frequently asked questions

Does blocking AI bots hurt my regular SEO?

No, as long as you only block AI bots and let Googlebot access your site normally. The two crawl systems are independent. Google-Extended is separate from Googlebot: blocking it has no effect on Google Search or AI Overviews. Be careful with AI search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot), though: blocking them does not affect your SEO, but removes you from those engines' answers.

Should I allow Common Crawl?

Common Crawl is the dataset on which many open-source LLMs are trained. Allowing it increases your chances of being in the training data of future models. If you don't have a specific reason to block it (sensitive content, paywall), allow it.

What happens if I block AI bots after being indexed?

Information already in LLM training data stays — you can't "erase" it from the training corpus. However, for real-time AI search systems (Perplexity, ChatGPT search), blocking their search crawler (PerplexityBot, OAI-SearchBot) will prevent future citations. Blocking only a training crawler such as GPTBot does not.

→See all GEO criteria and their impact on your score