robots.txt — The Established Gatekeeper

The Robots Exclusion Protocol began in 1994 as a practical convention devised by Martijn Koster. In 2022 it became RFC 9309, an official IETF Standards Track specification.

A website places robots.txt at the root of its domain. The file groups instructions by user agent and uses rules such as Allow and Disallow to indicate which paths a crawler may request. These instructions are voluntary: they guide cooperative crawlers but do not technically prevent access. Nor does robots.txt determine how information obtained elsewhere may be stored, licensed or used.

Adoption is extensive. OpenAI documents separate controls for GPTBot, used for potential model training, and OAI-SearchBot, used for search. Anthropic, Google and Perplexity also publish crawler identities and robots.txt instructions.

This makes robots.txt the most established messenger in the experiment. A request for the file shows that a bot looked for its instructions. It does not by itself prove that the bot followed them—or what it did with any content it subsequently accessed.