Paste your robots.txt and instantly see which AI crawlers are blocked or allowed. GPTBot, ClaudeBot, PerplexityBot, CCBot and more — know your status in seconds.
Paste your robots.txt on the left and click Check to see which AI crawlers you are blocking or allowing.
Which AI crawlers does this tool check?
This tool checks for GPTBot (OpenAI/ChatGPT), ClaudeBot (Anthropic), PerplexityBot (Perplexity AI), CCBot (Common Crawl, used for training data), Google-Extended (Google's AI training opt-out), Applebot-Extended (Apple AI), and OAI-SearchBot (OpenAI real-time search). These are the most impactful AI crawlers for brand visibility in 2025 and 2026.
Should I block or allow AI crawlers?
For AI visibility, you generally want to allow AI crawlers. Blocking GPTBot means ChatGPT cannot crawl your site for real-time answers. Blocking PerplexityBot means Perplexity will not include your pages in cited responses. The only crawler you might selectively block is CCBot if you do not want your content used for training datasets, which is separate from real-time retrieval. See our GEO guide for the full strategy.
How do I allow a specific AI crawler?
Remove the specific User-agent: GPTBot / Disallow: / block from your robots.txt. Or if you have a wildcard block (User-agent: * / Disallow: /), add an explicit allow block: User-agent: GPTBot / Allow: / after it. Then re-upload your robots.txt.
My robots.txt allows all bots. Am I fully visible in AI results?
robots.txt is necessary but not sufficient. AI visibility also depends on your llms.txt file, schema markup, off-site citations, and content quality. Check our AI Visibility Checker for a fuller picture.