robots.txt is a plain-text file placed at the root of a website that tells web crawlers which pages or directories they may access — a voluntary convention, not a legal enforcement mechanism.
Part of the 📝Robots Exclusion Protocol (REP) first proposed by Martijn Koster in 1994, the file addresses directives to specific user agents (crawler identifiers) through Allow and Disallow rules, and must be served at the exact path /robots.txt on the root domain. Crawlers are expected to check it before fetching other pages; a missing file is read as "everything is allowed," and nothing prevents a crawler from ignoring it.
Originally designed to manage search-engine indexing, robots.txt has been repurposed as the primary mechanism for signaling whether content may be used as 📝training data for AI models. 📝OpenAI's GPTBot, 📝Anthropic's ClaudeBot, Google-Extended, and Common Crawl's CCBot can each be blocked by name — currently the most widely adopted opt-out. The mechanism is limited: it is not legally binding, cannot undo prior scraping, offers no granularity (there is no way to say "index for search but don't train"), and leaves no audit trail of compliance.
It is a 1994 solution stretched to a 2026 problem. Initiatives like the Do Not Train registry and machine-readable licensing standards aim to build proper consent infrastructure for 📝Large Language Models (LLMs); none has reached industry-wide adoption, so robots.txt remains the default — a voluntary signal in a space that increasingly demands enforceable boundaries.
📝MythOS serves its own robots.txt, and configuring it surfaced the tension: blocking AI crawlers protects content from unconsented training, but the platform also distributes content through AI — via 📝MCP, llms.txt, and the chat API. robots.txt is a binary in a world that needs permissions.