Content Signals in robots.txt: declare search, AI input, and training preferences
Content Signals let you publish machine-readable preferences for search, AI input, and training in robots.txt — separate from classic Allow/Disallow crawler rules. Use them to state intent clearly without confusing retrieval access with training policy.
Last updated: 23 August 2026
Learn what each signal means, how it differs from blocking GPTBot, and how to verify crawl access on a live URL.
User-agent: *Allow: /Content-Signal: search=yes, ai-input=yes, training=noUser-agent: GPTBotAllow: /Why Content Signals are not the same as blocking bots?
Allow/Disallow decides whether a crawler may fetch a URL. Content Signals express preferences about how content may be used for search, AI input, and training. Mixing the two without a policy creates accidental invisibility in AI answers.
Retrieval access
User-agent rules for Googlebot, GPTBot, ClaudeBot, and similar agents control whether pages can be fetched for ranking or answering.
Usage preferences
Content Signals (search / AI input / training) communicate intended use. They complement — and do not replace — careful crawler Allow rules.
Checklist before you publish Content Signals
Decide policy first, then encode it. Do not copy a snippet that blocks the visibility you want.
Have you separated retrieval policy from training policy?
Want citations? Keep retrieval bots allowed. Training preferences can differ — do not Disallow GPTBot solely because you dislike training.
Do search, ai-input, and training values match leadership intent?
Document yes/no for each signal with legal/marketing alignment before editing production robots.txt.
Is robots.txt still valid plain text with a 200 response?
Broken syntax or HTML error pages mean crawlers may ignore your preferences entirely.
Did you keep Googlebot healthy?
Content Signals should not come with accidental sitewide Disallow or missing Sitemap lines.
Are CMS/security plugins overwriting robots.txt?
Many plugins regenerate the file and drop custom Content-Signal lines after updates.
Will hosted or subdomain pages inherit the same policy?
If you publish elsewhere, confirm which robots.txt origin applies to that host.
Have you re-fetched robots.txt after deploy?
Verify the live file shows the signals you intended — then re-check a money URL for SEO/GEO access.
Inseo flags blocked AI crawlers and weak discovery files alongside SEO/GEO scores.
Common Content Signal mistakes
Use this table when a “privacy” robots edit quietly killed AI visibility.
| Mistake | Effect on search | Effect on AI answers |
|---|---|---|
| Disallow GPTBot “to be safe” | Google may still rank | Answer engines cannot retrieve |
| Signals without Allow rules | Ambiguous intent | Preferences ignored if fetch blocked |
| Plugin overwrite | Sitemap line may vanish | Custom signals disappear overnight |
| training=no ≠ block retrieval | Usually unchanged | Confusion leads to over-blocking |
| HTML robots “page” | Crawlers may ignore | No reliable preferences |
| Different policy per host | Main domain ok | Subdomain/offer host disagrees |
How to declare signals without hurting visibility
Write policy in plain language first, then encode the minimum clear rules.
What do search, ai-input, and training mean in practice?
They are preference labels for how you want content used. Pair them with explicit Allow/Disallow for the bots that matter to ranking and answering.
Can I allow AI answers but disallow training?
Often that is the intent — keep retrieval agents allowed for citation, and set training preferences separately. Confirm current bot behavior and legal guidance for your region.
Should every site publish Content Signals?
Publish them when you have a real policy. An empty or contradictory robots.txt is worse than a simple Allow with a documented stance.
How do I verify the change worked?
Fetch /robots.txt, confirm syntax and values, then run a live URL check for crawler access and SEO/GEO readiness.
How Inseo helps
Catch robots mistakes before they hide you from AI answers
Inseo reviews crawl access and SEO/GEO signals on your live URL so you can see whether Content Signals and bot rules still allow the visibility you want — then generate a blueprint when the page itself needs restructuring.
- Surface blocked AI crawlers and discovery-file gaps early
- Keep search foundations and GEO extractability on the same audit
- On Enterprise, hosted pages can ship with coherent robots / llms discovery files
Questions about Content Signals
Are Content Signals a Google ranking factor?
Treat them as preference declarations, not a magic ranking boost. Healthy crawl access and strong pages still matter most.
If I set training=no, will ChatGPT stop citing me?
Not automatically. Citation depends on retrieval access and page quality. Blocking retrieval bots is what usually removes you from answers.
Do Content Signals replace llms.txt?
No. llms.txt helps models find priority URLs. Content Signals express usage preferences in robots.txt.
Can Inseo invent a legal policy for my company?
No. You choose the policy. Inseo helps you see technical side effects on crawl access and page readiness.
Related guides
Step-by-step how-tos and deeper articles for the gaps this page diagnoses.