GPTBot, ClaudeBot, PerplexityBot — which AI crawlers matter for your visibility
For GEO, prioritize retrieval bots that fetch pages for answering — such as GPTBot, ClaudeBot, and PerplexityBot — separately from optional training crawlers. Blocking every AI user-agent can protect training data while quietly removing you from answer candidate sets.
Last updated: 23 August 2026
Learn which bots affect AI answers vs training, then configure robots.txt so retrieval can reach the pages you want cited.
User-agent: GPTBotAllow: /User-agent: ClaudeBotAllow: /User-agent: PerplexityBotAllow: /Why “block all AI bots” is the wrong default?
Many site owners copy a blanket Disallow for AI user-agents. That can stop unwanted training crawls — and also stop the fetchers that assemble ChatGPT, Claude, and Perplexity answers.
Training-oriented crawls
Some agents primarily collect data for model training. Teams often restrict these while still wanting to appear in answers.
Retrieval / answer crawls
Other agents fetch pages at answer time (or to build answer indexes). Blocking them removes you from many citation candidate sets even if Googlebot is fine.
AI crawler checklist for GEO
Separate retrieval access from training preferences.
Can GPTBot fetch your high-intent URLs?
Confirm robots.txt does not Disallow the paths you want cited in ChatGPT-style answers.
Is ClaudeBot allowed where you want Claude citations?
Same pattern: allow retrieval on service and location pages you care about.
Is PerplexityBot allowed?
Perplexity relies on live retrieval; blocked paths rarely appear as sources.
Did you avoid a blanket AI Disallow?
Wildcard blocks for all AI agents often catch retrieval bots you still need.
Are Googlebot and Bingbot still open?
GEO does not require sacrificing classic SEO crawl access.
Have you reviewed Content Signals / Google-Extended?
Training and preview preferences are separate from “can this page be fetched for answers.” Read vendor docs before copying rules.
Do logs show the bots you expect?
After allowing, verify user-agents in access logs or WAF rules that might still block them.
Inseo flags blocked retrieval bots as part of the GEO readiness check.
Crawler mistakes that kill AI visibility
Technical defaults that look “secure” but hide you from answers.
| Config | Google often ok | AI answers fail |
|---|---|---|
| Disallow: / for GPTBot | Googlebot unaffected | ChatGPT cannot fetch |
| WAF bot challenge | Humans pass | Fetchers get blocked |
| JS-only body | May render for Google | Many AI fetchers see shells |
| Copy-paste “block AI” gist | Feels modern | Retrieval agents blocked too |
| Staging robots on prod | Accident | Everything disallowed |
| Ignore PerplexityBot | Not a Google issue | Perplexity sources skip you |
How to choose which bots to allow
Decide per goal — answers vs training — then encode it in robots.txt.
Must I allow every AI bot?
No. Allow the retrieval agents for the answer surfaces you care about. Restrict training crawlers deliberately if that is your policy.
Is Google-Extended the same as Googlebot?
No. Googlebot handles Search. Google-Extended relates to Gemini / grounding-style use — treat them as separate decisions.
Where should I start?
Allow GPTBot, ClaudeBot, and PerplexityBot on the URLs you want cited; keep Googlebot open; then refine training rules.
What about Content Signals?
Use them to declare search / AI input / training preferences when your stack supports them — do not confuse them with a total bot ban.
Come aiuta Inseo
Detect blocked AI crawlers on a live URL
Paste your page. Inseo checks SEO and GEO signals — including crawler access patterns — then can generate a blueprint that assumes retrieval bots can reach the content.
- Ready checks surface AI bot blocks
- Pairs with robots.txt and Content Signals guides
- Free preview first — generate when access is fixed
Questions about AI crawlers
Will allowing GPTBot put my content in training data?
Policies differ by vendor and change over time. Separate retrieval allow rules from training restrictions, and read the current publisher docs for each bot.
Does allowing bots guarantee citations?
No. Access is necessary, not sufficient. You still need quotable structure and relevance.
Should ecommerce block all AI bots?
Only if you accept disappearing from answer engines. Many teams allow retrieval on key category/product URLs while restricting training crawls.
Can Inseo edit my robots.txt for me?
Inseo diagnoses blocks on the URL you check. Hosted publish ships crawler-friendly defaults; root robots on your main domain stays under your control.
Guide correlate
How-to e articoli più approfonditi sulle lacune che questa pagina diagnostica.