How Should robots.txt Handle AI Crawlers?
Direct answer
Your robots.txt should deliberately permit the crawlers behind the visibility you want: Googlebot and Bingbot for search, OAI-SearchBot and ChatGPT-User for ChatGPT answers, PerplexityBot, ClaudeBot, and Google-Extended and GPTBot if you accept training use. The most common failure is an old wildcard block silently excluding all of them; the fix is an explicit, reviewed policy.
- Which AI crawlers matter and what does each do?
- How do accidental blocks happen?
- What does a sane policy look like?
- Frequently asked questions
Which AI crawlers matter and what does each do?
Search-answer crawlers fetch pages for live answers; training crawlers collect content for model training; they deserve separate decisions.
OAI-SearchBot powers ChatGPT search results and ChatGPT-User fetches pages users ask about: blocking them removes you from ChatGPT answers. GPTBot and Google-Extended feed model training: blocking them is a legitimate philosophical choice with a visibility cost, since trained knowledge is one of the two supplies answers draw from. PerplexityBot and ClaudeBot serve their platforms' retrieval similarly.
How do accidental blocks happen?
Through legacy wildcard rules, security plugins, and CDN bot protection acting without anyone deciding anything.
A `User-agent: *` disallow added during development and never removed, a WordPress security plugin's aggressive bot list, or CDN-level bot management defaulting to challenge-everything: each silently excludes AI crawlers. The tell is a business with good content and zero AI presence; the diagnosis takes one look at robots.txt and the CDN settings.
What does a sane policy look like?
Explicit allows for the crawlers you want, considered decisions on training bots, and a sitemap line, reviewed annually.
List the search and answer crawlers with explicit Allow rules so future wildcard edits cannot catch them, decide training-bot access deliberately, point to your sitemap, and calendar an annual review. Our own robots.txt permits Googlebot, Bingbot, OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Applebot, and Google-Extended, a visibility-first policy we recommend for most businesses.
robots.txt review checklist
- No wildcard rule silently blocking AI crawlers
- Search-answer bots explicitly allowed
- Training-bot access decided deliberately, not by default
- CDN and security-plugin bot rules checked too
- Sitemap referenced; annual review scheduled
Common mistakes to avoid
- Assuming robots.txt is fine because nobody has touched it
- Blocking training bots while wondering why models do not know you
- Fixing robots.txt but leaving CDN bot-protection challenges in place
Frequently asked questions
Where do I check what my robots.txt says?
At yourdomain.com/robots.txt in any browser; server logs then confirm which bots actually reach you, since CDN rules can override what the file implies.
Is there any reason to block AI search crawlers?
Publishers with content-licensing concerns sometimes do; for a local business whose goal is being found and recommended, blocking them is self-defeating.
Last reviewed: July 10, 2026. We keep resource content maintained as AI platforms evolve.
Ready to find out what AI search tools say about your business?
Book a free introductory call to discuss your current AI visibility and determine the most appropriate next step.