Robots Exclusion Protocol Extension to manage AI content use

While the Robots Exclusion Protocol enables service owners to control how, if at all, automated clients known as crawlers may access the URIs on their services as defined by [RFC8288], the protocol doesn't provide controls on how the data returned by their service may be used in training generative AI foundation models. Application developers are requested to honor these tags. The tags are not a form of access authorization however.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

The possible values of the rules complementing existing allow, disallow rules are: DisallowAITraining - instructs the parser to not use the data for AI training language model. AllowAITraining - instructs the parser that the data can be used for AI training language model. The values are case insensitive and honor the same matching logic as Allow and disallow rules. When Allow and Disallow rules define if the content can be downloaded, AllowAITraining and DisallowAITraining rules only apply rules on usage of the content for AI training.

The same rules can also be set in the Application Layer Response Header: DisallowAITraining - instructs the parser to not use the data for AI training language model. AllowAITraining - instructs the parser that the data can be used for AI training language model. The values are case insensitive and honor the same matching logic as Allow and disallow rules.

Same rules can also be set via an HTML meta tag: <meta name="robots" content="DisallowAITraining"> <meta name="examplebot" content="AllowAITraining">

TODO: https://www.rfc-editor.org/rfc/rfc9110.html#name-field-name-registry