Are AI crawlers blocked on your site?
To see if AI crawlers are blocked, inspect robots.txt for GPTBot, ClaudeBot, and PerplexityBot, then verify live fetches are not stopped by WAF or CDN rules that allow Googlebot.
If retrieval bots cannot fetch the page, you will not appear in many AI answers — even when Googlebot is fine. Verify robots.txt and edge rules.
Retrieval vs training
How to verify
robots.txt rules
Note User-agent blocks and path Disallows that cover your high-intent URLs.
WAF / bot challenges
Edge rules may return challenges that humans pass and fetchers fail — check access logs for those user-agents.
Live fetch test
Request the URL with a tool that can set User-Agent (or your CDN logs) and confirm 200 with HTML body, not an empty challenge page.
What to check on your page
Is Googlebot still allowed?
GEO should not sacrifice classic SEO crawl access.
Did you paste a viral “block AI” gist?
Many gists Disallow retrieval agents you still need.
FAQ
Is Google-Extended the same as Googlebot?
No. Treat Search crawl and training/grounding preferences as separate decisions.
Does allowing a bot guarantee citation?
No. Access is necessary, not sufficient.