73% of websites block AI bots — GPTBot, ClaudeBot, PerplexityBot — without their owners knowing (Otterly.AI, 2026). In practice, this means ChatGPT, Claude, and Perplexity simply cannot read the content on these sites. And content that cannot be read will never be cited.

Most of the time, the blocking is not intentional. It comes from a misconfigured robots.txt file, a CMS that blocks bots by default, or a site entirely rendered in client-side JavaScript. The result is the same: you are invisible to AI engines, even if your content is excellent.

Here is how to check if you are affected, why this is a critical problem in 2026, and how to fix it in under 10 minutes.

Why AI bots are blocked by default

Historically, the robots.txt file was used to control access for Google and Bing crawlers. When AI bots appeared (GPTBot in August 2023, ClaudeBot, PerplexityBot), many websites and CMS platforms reacted by blocking them as a precaution.

The problem: this decision was often made at the CMS or hosting level, not by the site owner. If you use WordPress with certain security plugins, Wix, Squarespace, or a CDN like Cloudflare with aggressive anti-bot rules, there is a strong chance AI bots are already blocked on your site.

The three main causes:

  • Explicit robots.txt: lines like User-agent: GPTBot / Disallow: / deny access to your entire site. This is the most common case and the simplest to fix.
  • Security plugins: some WordPress plugins (Wordfence, Sucuri, iThemes Security) or Cloudflare rules block unrecognized user-agents, including AI bots.
  • Client-side JavaScript rendering: if your site is a Single Page Application (React, Angular, Vue) without Server-Side Rendering (SSR), AI bots see nothing but a blank page. Technically, the bot is not blocked, but it cannot read anything.

How to check if your site blocks AI bots

This is quick to verify. Here are the 3 steps:

Step 1: Check your robots.txt

Type your-website.com/robots.txt in your browser. Look for mentions of these user-agents:

  • GPTBot — the ChatGPT/OpenAI crawler
  • OAI-SearchBot — OpenAI's web search crawler
  • ClaudeBot — Anthropic's (Claude) crawler
  • PerplexityBot — Perplexity's crawler
  • Google-Extended — Google's AI crawler (distinct from Googlebot)
  • Bytespider — ByteDance's crawler (used by Chinese models)

If you see Disallow: / next to any of these agents, your site is inaccessible to them.

Step 2: Test your site's rendering

Open your site in Chrome, right-click, and select "View Page Source" (not the inspector — the raw source code). If you see readable content (text, paragraphs, headings), you are good. If you see mostly JavaScript and empty <div id="root"></div> tags, your content is not accessible to bots.

Step 3: Run an automated audit

Detekia automatically checks your site's crawlability by AI bots. The "AI Crawlability" criterion in our GEO score detects robots.txt blocks, the absence of an llms.txt file, indexation issues, and whether AI bots are present in your configuration.

Check if AI bots can access your website — free, under 60 seconds.

Test my website for free →

The 6 AI bots you need to know

Not all AI engines use the same crawler. Here are the main ones and their impact:

  • GPTBot (OpenAI) — powers ChatGPT and its web-search-enabled responses. This is the most important one: ChatGPT processes 2.5 billion queries per day. Blocking it makes you invisible to 810 million daily users.
  • OAI-SearchBot (OpenAI) — the crawler specific to ChatGPT's web search feature. Distinct from GPTBot, it is sometimes blocked separately.
  • ClaudeBot (Anthropic) — the crawler for Claude. Claude is used by millions of professionals and its API powers numerous B2B applications.
  • PerplexityBot — Perplexity's crawler. Perplexity is the AI search engine that systematically cites its sources with links. Being indexed by Perplexity generates direct traffic.
  • Google-Extended — controls the use of your content by Gemini and Google AI Overviews. Important: blocking Google-Extended does NOT block Googlebot (your standard SEO remains unaffected).
  • Applebot-Extended — used by Apple Intelligence and Siri. Relevant if your audience is on the Apple ecosystem.

What you lose by blocking AI bots

Blocking AI bots is not a neutral decision. Here is what it costs you in concrete terms:

Zero citations in AI responses. When a prospect asks ChatGPT "what is the best tool for [your domain]," your site cannot be cited if GPTBot cannot read it. Your competitors who allow access will be recommended instead.

Absent from Google AI Overviews. Google uses Google-Extended to power its AI summaries. If you block it, you can still appear in classic results, but never in the AI summary at the top of the page — the one that 83% of users read before clicking.

Lost AI-referred traffic. AI-referred traffic surged by 527% in 2025 (Previsible). Visitors from an AI recommendation convert 4.4x better than standard organic visitors (Semrush, 2025). That is qualified traffic you are handing to your competitors.

Growing invisibility. Gartner forecasts a 25% decline in traditional search volume by the end of 2026. The share of traffic migrating to AI will only increase. The longer you wait to unblock access, the wider the gap becomes.

→Why your Google traffic is dropping in 2026 (and what AI has to do with it)

How to unblock access in 10 minutes

1. Edit your robots.txt

If your robots.txt contains blocking directives, replace them. Here is an example configuration that allows all AI bots:

# Allow AI bots
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

If you want to block certain sensitive pages (client area, admin pages) while allowing the rest:

User-agent: GPTBot
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /api/

2. Create an llms.txt file

The llms.txt file is an emerging standard that tells AI engines how to interpret your site. Place it at the root (your-website.com/llms.txt):

# My Company
> Short description of what your company does.

## Main pages
- [Home](https://your-website.com/)
- [Product](https://your-website.com/product)
- [Blog](https://your-website.com/blog)
- [About](https://your-website.com/about)
- [Contact](https://your-website.com/contact)

## What we do
Clear, factual description of your business,
your services, and your value proposition.

This file helps AI engines understand your site's structure and identify the most important pages. It is the equivalent of a sitemap for AI engines.

→Complete guide: llms.txt, robots.txt, and AI crawlability

3. Check security plugins and rules

If you use WordPress:

  • Wordfence: go to Firewall → Rate Limiting. Verify that AI user-agents are not on the block list.
  • Sucuri: in the dashboard, check the bot blocking rules.
  • Rank Math / Yoast: these plugins do not block AI bots by default, but check the robots.txt they generate automatically.

If you use Cloudflare:

  • Go to Security → WAF → Custom Rules.
  • Verify that no rule blocks the GPTBot, ClaudeBot, or PerplexityBot user-agents.
  • In Security → Bots, make sure "Bot Fight Mode" is not too aggressive — it can block legitimate AI crawlers.

4. Fix JavaScript rendering issues

If your site is a SPA (Single Page Application):

  • Ideal solution: switch to Server-Side Rendering (SSR) with Next.js, Nuxt.js, or similar. Content is rendered server-side and immediately readable by bots.
  • Intermediate solution: use a pre-rendering service (Prerender.io, Rendertron) that serves a static HTML version to bots.
  • Verification: test your site with curl -s your-website.com | head -50. If you see text content, you are good. If it is pure JavaScript, bots see nothing.

Special case: should you allow ALL bots?

This is a fair question. Some sites choose to block specific bots for strategic reasons (protecting exclusive content, sensitive data, etc.).

Our recommendation:

  • Allow GPTBot, ClaudeBot, PerplexityBot, and Google-Extended unless you have a documented reason not to. These 4 bots cover 95% of AI traffic.
  • Selectively block sensitive pages (admin, client area, API) rather than the entire site.
  • Review your decision quarterly. The AI ecosystem evolves fast. New bots appear, others become dominant.

Blanket blocking no longer protects your content — AI engines can find your information through other sources (caches, aggregators, third-party mentions). What it does guarantee is that you will never be cited as a source, which is the worst-case scenario.

Check your crawlability now

AI crawlability is one of the 8 criteria in the Detekia GEO score. It is also the simplest criterion to fix: a 2-line change in your robots.txt may be all it takes to unlock your AI visibility.

Detekia automatically analyzes your robots.txt file, detects the presence or absence of an llms.txt, and verifies whether your content is accessible to AI crawlers. The diagnostic is free and takes under 60 seconds.

→The 8 GEO criteria that determine if AI cites you — Detekia methodology

Frequently asked questions

Does unblocking AI bots pose a security risk?

No. AI bots read the public content on your site, exactly as Googlebot has done for 20 years. They do not access your databases, your admin panel, or protected data — as long as your robots.txt denies access to sensitive directories (/admin/, /api/, etc.).

Will my content be used to train AI models?

This is a separate question. GPTBot is used for both real-time web search (RAG) and potentially for training. If you want to allow web search but not training, you can allow OAI-SearchBot while blocking GPTBot. For Google, Google-Extended controls only AI usage — blocking it does not affect your standard SEO.

How long before AI engines start citing me?

After unblocking access, AI bots recrawl your site within a few days to a few weeks. The first citation results typically appear in 4 to 8 weeks, depending on your content quality and sector competitiveness. Crawlability is a necessary condition, not a sufficient one — your content also needs to be extractable, sourced, and structured.

→GEO: the complete guide to being cited by AI engines in 2026

Key takeaways

73% of sites block AI bots without knowing it. If that includes you, no other GEO optimization will work until the block is lifted. It is the first thing to check, and often the simplest to fix.

Check your robots.txt, create an llms.txt file, disable overly aggressive security rules, and make sure your content is server-side rendered. In 10 minutes, you go from invisible to indexable by every AI engine.

Is your site blocking AI bots? Check for free in under 60 seconds.

Test my website for free →