Sitemaps & Robots are the library-wide signals that tell crawlers what to index and how to behave. As an 📝SEO & AEO feature, MythOS generates sitemaps covering your public memos, tags, and communities, plus a robots.txt that welcomes search and AI crawlers while keeping private surfaces out. It exists so search engines find everything worth indexing and nothing that isn't, with no files to write or maintain. It's for every creator who wants reliable indexing without touching configuration.
Key Capabilities
- Sitemap index — a root sitemap links child sitemaps for memos, tags, communities, and community posts
- Per-creator sitemaps — each eligible creator gets a sitemap of their own public memos
- Crawler-aware robots.txt — allows major search and AI bots while disallowing private routes
- AI crawler allowlist — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more are explicitly allowed
- Scraper controls — abusive crawlers such as Bytespider are disallowed
Getting Started
Sitemaps and robots.txt generate automatically — there is nothing to configure. Publish public memos and they appear in your sitemaps; the platform robots.txt at the site root advertises your sitemap and llm.txt to crawlers. Per-creator sitemaps are available on indexable tiers. You can submit your sitemap URL to a search console for faster indexing, but crawlers also discover it through robots.txt.
FAQ
- Do I need to submit a sitemap? No — sitemaps generate automatically and robots.txt advertises them. Submitting to a search console can speed up indexing but isn't required.
- Which AI crawlers are allowed? Major answer-engine bots — GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended and others — are explicitly allowed.
- Are any crawlers blocked? Yes — abusive scrapers like Bytespider are disallowed, and private routes such as settings, auth, and API are excluded from all crawlers.
- What's in the sitemaps? Public, non-trashed memos by claimed creators, plus tag, community, and community-post pages.