When robots.txt or llms.txt quietly hurt visibility
Visibility can drop overnight when a staging robots.txt ships to production, a Disallow rule expands, or llms.txt still points at deleted URLs. Treat those files as living inventory.
A well-meaning “block AI” gist, a staging robots file on production, or an outdated llms.txt can erase citation readiness overnight.
Visibility drift from config files
How to monitor
Diff robots.txt in CI
Fail builds when Disallow rules for known retrieval agents appear unexpectedly.
Re-fetch after CDN or WAF changes
Edge config is a common silent breaker.
Review llms.txt when IA changes
Update links when you ship new service or location URLs.
What to check on your page
Compare live robots.txt to last month’s copy
Look for new AI Disallows.
Open every llms.txt URL
Remove 404s and redirects to irrelevant pages.
FAQ
Can sitemap drift cause the same issue?
Yes — missing URLs slow discovery — but robots Disallow is a harder block for AI retrieval.
Should marketing own these files?
Someone must. Treat them like production config, not a one-time SEO task.