Robots Exclusion Protocol (REP) is the convention by which a website publishes, in a file named 📝robots.txt, rules that automated crawlers are requested to honor when accessing its URLs, standardized by the IETF as RFC 9309.
RFC 9309, published in September 2022 as an Internet Standards Track document of the Internet Engineering Task Force, extends the method Martijn Koster originally defined in 1994 for service owners to control how crawlers access their content. Its authors are M. Koster, G. Illyes, H. Zeller and L. Sassman, the last three of Google LLC, and it adds definition language plus instructions for handling errors and caching.
Under the standard, rules sit in groups that begin with user-agent lines naming a crawler's product token, followed by allow and disallow rules matched against URI paths, with the most specific match used. The file must be served as /robots.txt in the top-level path of a service, UTF-8 encoded, as text/plain. Crawlers should not use a cached copy beyond 24 hours unless the file is unreachable.
The RFC states that these rules are not a form of access authorization and that the protocol is not a substitute for valid content security measures, since listing paths in robots.txt makes them discoverable. Google's documentation, which says its crawlers support the REP, describes robots.txt as used mainly to avoid overloading a site with requests, not as a mechanism for keeping a page out of Google, and notes that obeying it is up to each crawler.