Fix robots.txt and AI crawler access
Publish robots.txt at the site root. Allow retrieval crawlers if you want AI answers to see the page. Blocking training (Google-Extended) is a separate choice. Follow the steps below on your stack, then re-check the live URL.
Publish robots.txt at the site root. Allow retrieval crawlers if you want AI answers to see the page. Blocking training (Google-Extended) is a separate choice.
Wir merken uns das in diesem Browser, damit die nächste Anleitung denselben Stack verwendet.
At a glance
| Time | ~15 min |
|---|---|
| Category | GEO |
| Steps | 3 |
| Stacks | WordPress, Shopify, Webflow, Next.js, HTML, more |
Schritte
Publish /robots.txt at the domain origin if it is missing.
Start with Allow: / and a Sitemap line.
Look for Disallow on GPTBot, ClaudeBot, PerplexityBot, or Google-Extended. Allow retrieval bots if you want citation.
In HTML / static
Serve /robots.txt at the origin with Allow rules for retrieval crawlers.
User-agent: * Allow: / Sitemap: https://yoursite.com/sitemap.xml User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: /
FAQ
- What does “Fix robots.txt and AI crawler access” fix?
- Publish robots.txt at the site root. Allow retrieval crawlers if you want AI answers to see the page. Blocking training (Google-Extended) is a separate choice.
- How long does this usually take?
- About 15 minutes on most CMS and code stacks when you already have access to edit the page.
- Where do I start?
- Start with: Publish /robots.txt at the domain origin if it is missing. Then use the platform picker on this page for placement notes.