Mythos

Split Sitemap Strategy is a crawler-routing tactic that sorts a site's content into classes and gives each class its own per-bot indexing rules, so pages built for AI answer engines stay open to AI crawlers while staying out of Google's search index.

The tactic exists because answer engines and search engines now reward opposite things. Fact-anchored best-of lists earn strongly in 📝Answer Engine Optimization (AEO) and read as thin or spammy to 📝Search Engine Optimization (SEO). Rather than choosing one, a site places that content in its own directory, disallows or noindexes it for Googlebot only, and leaves it fully crawlable for OAI-SearchBot, GPTBot, ClaudeBot, and the Google-Extended token that governs Gemini training and grounding. The name comes from the discovery half: the search sitemap lists only the standard class, while the AI-facing 📝LLMs.txt points at the AEO class. 📝Ryan Edwards of 📝CAMINO5 described the pattern in August 2026 as something large publishers already run and that 📝Cloudflare is turning into a default-configurable control.

How It Works

  • Content classes — Every page carries a class: a standard class served to search and AI alike, and an AEO-only class for list and fact content. The class is a configuration value, not something baked into the page, so a change in how any engine ingests is a one-line class edit rather than a content migration.
  • Per-bot crawl rules — Groups in robots.txt are keyed by user agent, and Google honors only the most specific group that matches a crawler while ignoring the rest. A Googlebot group can therefore Disallow the AEO directory while the wildcard group leaves it open to every other bot.
  • Per-bot index rules — The X-Robots-Tag HTTP header accepts a user-agent prefix, so X-Robots-Tag: googlebot: noindex, nofollow binds only Googlebot and a rule with no prefix binds every crawler. The same scoping exists in HTML as <meta name="googlebot" content="noindex">.
  • Crawl and index are separate controls — Google cannot read a noindex on a page it is forbidden to fetch, and a disallowed URL can still be indexed from external links as a bare URL. To keep the AEO class fully out of Google's index, let Googlebot crawl it and serve the user-agent-scoped noindex, rather than relying on Disallow alone.
  • Split discovery — The Sitemap line in robots.txt is not tied to any user-agent group, but Google follows it only when the sitemap URL is not disallowed for crawling. A Googlebot group that disallows the AEO sitemap's URL keeps Google from fetching it while every other bot follows the same line. Submit that sitemap to Bing Webmaster Tools and IndexNow, never to Search Console, since ChatGPT's retrieval leans on Bing's index. Do not route discovery through llms.txt: 2026 log studies show AI crawlers almost never request it.
  • Bot roster — Googlebot (📝Google search); Google-Extended (📝Gemini training and grounding, a control token only, with no separate user agent and no effect on Search inclusion or ranking); GPTBot (📝OpenAI training); OAI-SearchBot (📝ChatGPT search, and a site that opts out is not shown in ChatGPT search answers); ChatGPT-User (user-initiated fetches, where robots rules may not apply); ClaudeBot (📝Anthropic training); Claude-SearchBot (Claude search); Claude-User (user-initiated); PerplexityBot. Unlisted AI bots default to open on the AEO class.
  • Verification — After deploy, fetch each class as each bot and confirm the served rule matches the matrix, since one wrong group ordering silently inverts the policy.

Why It Matters

One crawler's trust signal is another crawler's spam signal, and serving both from a single robots policy forces a choice that the class abstraction dissolves. The pattern also anticipates where infrastructure is heading. Since July 1, 2026, Cloudflare classifies AI bots into Search, Agent, and Training categories with separate policies per category, and from September 15, 2026, Training and Agent crawlers are blocked by default on pages that display ads while Search crawlers stay allowed. Multi-purpose crawlers such as Googlebot, Applebot, and BingBot are evaluated under both policies, so a site that blocks Training blocks them too even where Search is allowed. Cloudflare is extending its Content Signals format in robots.txt with a use parameter that expresses immediate, reference, and full usage levels. Per-class, per-bot treatment is becoming the norm rather than a publisher hack.

For an enterprise such as 📝Adobe, this is a domain-level decision rather than a product team's, because robots.txt, sitemaps, and response headers belong to whoever owns the platform.

FAQ

Is "split sitemap" the right name?

Loosely. Sitemaps only handle discovery, and the split is real but indirect: a sitemap is withheld from Googlebot by disallowing its URL in the Googlebot group and by submitting it only to Bing and IndexNow, while enforcement lives in per-bot robots groups and user-agent-scoped noindex headers. "Split bot policy" and "class-based crawler routing" are the precise names; split sitemap is the memorable one.

Does blocking Googlebot from a directory hurt the rest of the site?

No, provided the class is isolated by path. Robots rules are path-scoped, the primary sitemap keeps every standard-class URL, and the only leakage risk is Google indexing disallowed URLs from external links as bare entries, which the user-agent-scoped noindex pattern avoids.

Does Google-Extended affect Google Search rankings?

No. Google states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal; it governs only whether crawled content trains future Gemini models or grounds them at prompt time, and it has no separate user agent string.

Why do AEO pages get punished in SEO?

Pure list and fact content reads as thin to Google's quality systems, while answer engines run simple fan-out queries such as "official site" or "best X" and reward exactly that shape, especially when each ranking is anchored to a verifiable fact from a trusted source.

What are the risks?

Robots directives are voluntary, user-initiated agents may ignore them, and noindex is honored by AI search bots as well as Google, so an unscoped noindex removes a page from every engine at once. 📝YMYL categories carry manual-action risk regardless of class and stay gated before publish.

I scoped this for MythOS on August 19, 2026 as class-based routing with two classes to start: standard memos, and a directory of best-of lists ranked by verifiable facts. The constraint I hold is that nothing may fragment the primary sitemap for the standard class. It surfaced again on the Adobe call on September 1 as the Adobe-wide play that sits above anything Acrobat controls.

Related

Contexts

Created with 💜 by One Inc | Copyright 2026