CLOUDFLARE AI CRAWL CONTROL ยท BOT ACCESS TEST

Cloudflare AI Crawler Access & robots.txt Diagnostic Tool

Test whether Cloudflare Bot Fight Mode, Managed robots.txt, or AI Crawl Control policies are silently blocking OAI-SearchBot, PerplexityBot, ClaudeBot, or Googlebot.

Run Instant Live Edge Inspection

2026 Technical Specification & Pass Criteria

Crawler User-AgentEngine PoweredRecommended 2026 Setting
OAI-SearchBot & ChatGPT-UserChatGPT Search & Live CitationsAllow in robots.txt + Bypass Bot Fight Mode
PerplexityBotPerplexity Answer EngineAllow in robots.txt + Content-Signal: search=yes
ClaudeBotAnthropic Claude Web SearchAllow in robots.txt + Accept: text/markdown
Googlebot & Google-ExtendedGoogle Search & AI OverviewsNever set blanket Training=Block on multi-purpose bots

Frequently Asked Questions

Can blocking 'AI Training' in Cloudflare hurt my Google SEO?

Yes! Cloudflare evaluates multi-purpose crawlers like Googlebot and Bingbot using the most restrictive rule โ€” a blanket Training block can inadvertently restrict search indexing unless configured carefully.

How do I check if Managed robots.txt injected Disallow rules on my domain?

Enter your domain above โ€” GeoReady fetches your live edge /robots.txt and parses every User-agent block automatically.

Related Free GEO Diagnostic Tools