Firewalls, Cloudflare, JS: Why Your Site Blocks AI Bots

73% of websites have technical barriers blocking AI crawler access (GPTBot, ClaudeBot, PerplexityBot…): robots.txt rules, CDN restrictions or JavaScript-only rendering, often without their owners knowing (Otterly.AI, 2026). In practice, this means ChatGPT, Claude and Perplexity cannot properly read the content on these sites. And content that cannot be read will never be cited.

Most of the time, the blocking is not intentional. It comes from a misconfigured robots.txt file, a CMS that blocks bots by default or a site entirely rendered in client-side JavaScript. The result is the same: you are invisible to AI engines, even if your content is excellent.

Here is how to check if you are affected, why this is a critical problem in 2026 and how to fix it in under 10 minutes.

This article focuses on the often invisible technical causes: firewalls, Cloudflare rules, security plugins and JavaScript rendering. For the detailed configuration of the files themselves (which user-agents to allow in robots.txt, how to write an llms.txt), the reference guide is llms.txt, robots.txt and AI accessibility.

Why AI bots are blocked by default

Historically, the robots.txt file was used to control access for Google and Bing crawlers. When AI bots appeared (GPTBot in August 2023, ClaudeBot, PerplexityBot), many websites and CMS platforms reacted by blocking them as a precaution.

The problem: this decision was often made at the CMS or hosting level, not by the site owner. If you use WordPress with certain security plugins, Wix, Squarespace or a CDN like Cloudflare with aggressive anti-bot rules, there is a strong chance AI bots are already blocked on your site.

The three main causes:

• Explicit robots.txt: lines like User-agent: GPTBot / Disallow: / deny access to your entire site. This is the most common case and the simplest to fix.
• Security plugins: some WordPress plugins (Wordfence, Sucuri, iThemes Security) or Cloudflare rules block unrecognized user-agents, including AI bots.
• Client-side JavaScript rendering: if your site is a Single Page Application (React, Angular, Vue) without Server-Side Rendering (SSR), AI bots see nothing but a blank page. Technically, the bot is not blocked, but it cannot read anything.

How to check if your site blocks AI bots

This is quick to verify. Here are the 3 steps:

Step 1: Check your robots.txt

Type your-website.com/robots.txt in your browser. Look for mentions of these user-agents:

• GPTBot — OpenAI's crawler for model training
• OAI-SearchBot — OpenAI's web search crawler (ChatGPT search)
• ChatGPT-User — the agent that visits a page when a ChatGPT user asks it to
• ClaudeBot — Anthropic's (Claude) crawler
• PerplexityBot — Perplexity's crawler
• Google-Extended — Google's AI crawler (distinct from Googlebot)
• Bytespider — ByteDance's crawler (used by Chinese models)

If you see Disallow: / next to any of these agents, your site is inaccessible to them.

Step 2: Test your site's rendering

Open your site in Chrome, right-click and select "View Page Source" (not the inspector — the raw source code). If you see readable content (text, paragraphs, headings), you are good. If you see mostly JavaScript and empty <div id="root"></div> tags, your content is not accessible to bots.

Step 3: Test your site automatically

Detekia automatically checks your site's crawlability by AI bots. The "Readable by AI" prerequisite of our GEO score detects AI search bots blocked in robots.txt, noindex tags and content that only appears with JavaScript. If it is missing, the score is capped at 40.

The 7 AI bots you need to know

Not all AI engines use the same crawler. Here are the main ones and their impact:

• OAI-SearchBot (OpenAI) — the crawler that lets your pages appear in ChatGPT's web-search-enabled responses. This is the most important one for citations: ChatGPT processes 2.5 billion queries per day. Blocking it makes you invisible in ChatGPT search.
• GPTBot (OpenAI) — the crawler that collects content that may be used to train OpenAI's models. Distinct from OAI-SearchBot, it is not used for real-time search (OpenAI documentation).
• ChatGPT-User (OpenAI) — the agent used when a ChatGPT user asks it to visit a specific page. It acts on demand, not as an automatic crawler.
• ClaudeBot (Anthropic) — the crawler for Claude. Claude is used by millions of professionals and its API powers numerous B2B applications.
• PerplexityBot — Perplexity's crawler. Perplexity is the AI search engine that systematically cites its sources with links. Being indexed by Perplexity generates direct traffic.
• Google-Extended — controls the use of your content for training Gemini models and for grounding answers in Gemini Apps and Vertex AI. Important: blocking Google-Extended does NOT block Googlebot and has no effect on your presence in Google Search, AI Overviews included (Google documentation).
• Applebot-Extended — used by Apple Intelligence and Siri. Relevant if your audience is in the Apple ecosystem.

What you lose by blocking AI bots

Blocking AI bots is not a neutral decision. Here is what it costs you in concrete terms:

Zero citations in AI responses. When a prospect asks ChatGPT "what is the best tool for [your domain]," your site cannot be cited if OAI-SearchBot cannot read it. Your competitors who allow access will be recommended instead.

Absent from Gemini. Google-Extended does not affect AI Overviews (which rely on Googlebot), but blocking it prevents Gemini from using your content to ground its answers. You remain visible in Google Search, but you disappear from a major AI assistant.

Lost AI-referred traffic. AI-referred traffic surged by 527% between January and May 2025 (Previsible). The average AI search visitor is worth 4.4× more than a standard organic visitor, thanks to a higher conversion rate (Semrush, 2025). That is qualified traffic you are handing to your competitors.

Growing invisibility. Gartner forecasts a 25% decline in traditional search volume by 2026. The share of traffic migrating to AI will only increase. The longer you wait to unblock access, the wider the gap becomes.

→Why your Google traffic is dropping in 2026 (and what AI has to do with it)

How to unblock access in 10 minutes

1. Edit your robots.txt

If your robots.txt contains blocking directives, replace them. Here is an example configuration that allows all AI bots:

# Allow AI bots
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

If you want to block certain sensitive pages (client area, admin pages) while allowing the rest:

User-agent: GPTBot
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /api/

2. Create an llms.txt file (optional)

The llms.txt file is a proposed standard that introduces your site to AI engines. It is not required to be read or cited: unlike robots.txt, it does not unblock anything. Place it at the root (your-website.com/llms.txt):

# My Company
> Short description of what your company does.

## Main pages
- [Home](https://your-website.com/)
- [Product](https://your-website.com/product)
- [Blog](https://your-website.com/blog)
- [About](https://your-website.com/about)
- [Contact](https://your-website.com/contact)

## What we do
Clear, factual description of your business,
your services, and your value proposition.

This file lists your most important pages for AI engines, a bit like a sitemap. Its effect on citations has not been demonstrated, and Google states that no special file is required to appear in its AI Overviews. It is a low-cost extra, not a priority.

→Complete guide: llms.txt, robots.txt and AI accessibility

3. Check security plugins and rules

If you use WordPress:

• Wordfence: go to Firewall → Rate Limiting. Verify that AI user-agents are not on the block list.
• Sucuri: in the dashboard, check the bot blocking rules.
• Rank Math / Yoast: these plugins do not block AI bots by default, but check the robots.txt they generate automatically.

If you use Cloudflare:

• Go to Security → WAF → Custom Rules.
• Verify that no rule blocks the OAI-SearchBot, GPTBot, ClaudeBot or PerplexityBot user-agents.
• In Security → Bots, make sure "Bot Fight Mode" is not too aggressive — it can block legitimate AI crawlers.

Watch out: since July 1, 2025, Cloudflare blocks AI crawlers by default on the new domains it protects. If your site moved behind Cloudflare after that date, check this setting explicitly: your robots.txt may allow OAI-SearchBot or PerplexityBot while the CDN rejects them upstream.

4. Fix JavaScript rendering issues

If your site is a SPA (Single Page Application):

• Ideal solution: switch to Server-Side Rendering (SSR) with Next.js, Nuxt.js, or similar. Content is rendered server-side and immediately readable by bots.
• Intermediate solution: use a pre-rendering service (Prerender.io, Rendertron) that serves a static HTML version to bots.
• Verification: test your site with curl -s your-website.com | head -50. If you see text content, you are good. If it is pure JavaScript, bots see nothing.

Special case: should you allow ALL bots?

This is a fair question. Some sites choose to block specific bots for strategic reasons (protecting exclusive content, sensitive data, etc.).

Our recommendation:

• Allow OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot and Google-Extended unless you have a documented reason not to. These are the main bots of the most widely used AI engines.
• Selectively block sensitive pages (admin, client area, API) rather than the entire site.
• Review your decision quarterly. The AI ecosystem evolves fast. New bots appear, others become dominant.

Blanket blocking no longer protects your content — AI engines can find your information through other sources (caches, aggregators, third-party mentions). What it does guarantee is that you will never be cited as a source, which is the worst-case scenario.

Check your crawlability now

Being readable by AI crawlers is the prerequisite of the Detekia GEO score: if they are blocked in robots.txt, the score is capped at 40. It is also the simplest issue to fix: a 2-line change in your robots.txt may be all it takes to unlock your AI visibility.

Detekia automatically analyzes your robots.txt file, detects the presence or absence of an llms.txt, and verifies whether your content is accessible to AI crawlers. The diagnostic is free.

Unblocking access is the first step, not the last. To find out whether ChatGPT, Gemini, Claude or Perplexity actually cite you once the doors are open, and how often compared with your competitors, an AI visibility audit of your brand queries these engines directly on your key queries.

→The 6 Detekia score criteria and the readability prerequisite — methodology

Frequently asked questions

Does unblocking AI bots pose a security risk?

No. AI bots read the public content on your site, exactly as Googlebot has done for 20 years. They do not access your databases, your admin panel or protected data — as long as your robots.txt denies access to sensitive directories (/admin/, /api/, etc.).

Will my content be used to train AI models?

This is a separate question. At OpenAI, the two uses are distinct: OAI-SearchBot powers ChatGPT's real-time web search, while GPTBot is used for model training (OpenAI documentation). If you want to allow web search but not training, you can allow OAI-SearchBot while blocking GPTBot. For Google, Google-Extended controls only AI usage — blocking it does not affect your standard SEO.

How long before AI engines start citing me?

After unblocking access, AI bots recrawl your site after a variable delay, from a few days to a few weeks depending on the engine. For citations, there is no typical delay: it also depends on your content quality and sector competitiveness. Measure before, then at regular intervals (for example monthly), rather than waiting for a specific date. Crawlability is a necessary condition, not a sufficient one — your content also needs to be extractable, sourced and structured.

→GEO: the complete guide to being cited by AI engines in 2026

Key takeaways

73% of sites have technical barriers for AI bots, often without knowing it (Otterly.AI, 2026). If that includes you, no other GEO optimization will work until the block is lifted. It is the first thing to check, and often the simplest to fix.

Check your robots.txt, disable overly aggressive security rules and make sure your content is server-side rendered. In 10 minutes, you go from invisible to indexable by every AI engine.