How to configure robots.txt so AI crawlers can retrieve your pages (without hurting SEO)
robots.txt controls who may fetch your URLs. Blocking GPTBot or similar retrieval crawlers can hide pages from AI answers even when Googlebot is allowed and rankings look healthy.
Last updated: 23 August 2026
Audit Disallow rules for AI bots, keep Googlebot healthy, then verify with a live URL check.
User-agent: *Allow: /Sitemap: https://yoursite.com/sitemap.xmlUser-agent: GPTBotAllow: /User-agent: ClaudeBotAllow: /Why can Google rank you while AI bots are blocked?
Search crawlers and AI retrieval crawlers are different user-agents. A site can Allow Googlebot and Disallow GPTBot — rankings continue, citations stall.
SEO-safe baseline
Allow: / for Googlebot, a valid Sitemap line, and no accidental sitewide Disallow.
GEO-safe baseline
Allow known retrieval agents (or * with care). Treat training preferences (for example Google-Extended) as a separate policy choice.
robots.txt checklist for AI visibility
Work top to bottom. Fix access before rewriting copy.
Is /robots.txt live at the origin?
Fetch https://your-domain/robots.txt and confirm a 200 text response.
Are GPTBot / ClaudeBot / PerplexityBot disallowed?
Search for Disallow rules on those agents. Allow them if you want answer engines to retrieve pages.
Did a plugin block AI by default?
Security and “AI blocker” plugins often add blanket Disallows — review after installs.
Is Googlebot still allowed?
Do not “fix GEO” by blocking Googlebot. Keep classic crawl healthy.
Are training and retrieval confused?
Blocking training (Google-Extended) is a policy choice; blocking retrieval bots cuts answer visibility.
Does Content-Signal match intent?
If you publish Content Signals, ensure they align with how you want AI input vs training handled.
Inseo flags blocked AI crawlers in site checks, then you can regenerate or fix manually.
robots.txt patterns that hurt GEO
Common configurations that look intentional but silently kill citations.
| Rule | Google impact | AI answer impact |
|---|---|---|
| Disallow GPTBot | Usually none | ChatGPT retrieval may skip you |
| Disallow: / on * | Severe if it hits Googlebot | Blocks most fetchers |
| AI blocker plugin | Often fine for Google | Blanket Disallow on AI agents |
| Staging rules on prod | Indexation disasters | Same — nothing fetches |
| Allow but noindex pages | Pages drop from Google | Thin or empty answers |
How do you change robots.txt safely?
Prefer additive Allow rules for retrieval agents over deleting your whole file.
What is a safe starting file?
User-agent: * with Allow: /, a Sitemap line, then explicit Allow for GPTBot and ClaudeBot if you want citations.
Will allowing AI bots hurt SEO?
Allowing retrieval crawlers does not remove Google rankings. Blocking Googlebot does.
What about training vs answers?
Separate policies. You can limit training while still allowing retrieval — document the choice for legal and marketing stakeholders.
How do I verify the fix?
Re-fetch /robots.txt, confirm status codes, then run an Inseo check on a key URL.
How Inseo helps
Catch blocked AI crawlers before you rewrite copy
Paste a ranking URL. Inseo reviews crawl access alongside SEO and GEO structure so you fix the real blocker first.
- Site checks highlight robots and crawler access issues
- GEO mode strengthens extractable content once bots can fetch
- Free preview on your live URL
Questions about robots.txt and AI
Is blocking GPTBot required for privacy?
It is a policy choice. Blocking retrieval reduces answer visibility; discuss training vs retrieval with legal before blanket Disallows.
Does Allow: / for * cover AI bots?
Often yes, unless a more specific User-agent block overrides it. Always read the full file.
Can I allow bots only on some paths?
Yes — Allow specific sections while Disallowing private areas. Keep public answer pages reachable.
Where is the stack how-to?
See the robots.txt how-to in the knowledge base for WordPress, Shopify, and other platforms.
Related guides
Step-by-step how-tos and deeper articles for the gaps this page diagnoses.