Test whether Cloudflare Bot Fight Mode, Managed robots.txt, or AI Crawl Control policies are silently blocking OAI-SearchBot, PerplexityBot, ClaudeBot, or Googlebot.
| Crawler User-Agent | Engine Powered | Recommended 2026 Setting |
|---|---|---|
| OAI-SearchBot & ChatGPT-User | ChatGPT Search & Live Citations | Allow in robots.txt + Bypass Bot Fight Mode |
| PerplexityBot | Perplexity Answer Engine | Allow in robots.txt + Content-Signal: search=yes |
| ClaudeBot | Anthropic Claude Web Search | Allow in robots.txt + Accept: text/markdown |
| Googlebot & Google-Extended | Google Search & AI Overviews | Never set blanket Training=Block on multi-purpose bots |
Yes! Cloudflare evaluates multi-purpose crawlers like Googlebot and Bingbot using the most restrictive rule โ a blanket Training block can inadvertently restrict search indexing unless configured carefully.
Enter your domain above โ GeoReady fetches your live edge /robots.txt and parses every User-agent block automatically.