]> Open Future

paul@openfuture.eu

Mozilla

mt@lowentropy.net

Web and Internet Transport AI Preferences AI Preferences Opt-Out Vocabulary This document defines a vocabulary for expressing preferences regarding how digital assets are used by automated processing systems. This vocabulary allows for the declaration of restrictions or permissions for use of digital assets by such systems. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the AI Preferences Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction This document defines a vocabulary of preferences regarding how automated systems process digital assets -- in particular, the training and use of AI models. This vocabulary can be used to describe the types of uses that a declaring party may wish to explicitly restrict or allow. The vocabulary is intended to be used in jurisdictions where expressing preferences results in legal obligations, as well as where there are no associated legal obligations. In either case, expressing preferences is without prejudice to applicable laws, including the applicability of exceptions and limitations to copyright. defines the data model for AI Preferences. defines the terms of the vocabulary. explains how to use AI Preferences in a data processing application, and describes a way to serialize preferences into a string. describes a process for determining the preference for a category of use. defines mechanisms to associate preferences with assets. Other means of association might be defined separately in the future.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document uses the following terms:

Artificial Intelligence (AI):

An engineered system of sufficient complexity that, for a given set of human-defined objectives, learns from data to generate outputs such as content, predictions, recommendations, or decisions.

AI Training:

The application of machine learning to data to produce or improve a model for an artificial intelligence system.

Asset:

A digital file or stream of data, usually with associated metadata.

Declaring party:

The entity that expresses a preference with regards to an Asset.

Machine Learning (ML):

The processing of data to produce or improve a model that encodes the relationship between the data and human-defined objectives.

Search Application:

A search application is a system that enables users locate items on the internet or in a specific data store.

Statements of Preference The vocabulary is a set of categories, each of which is defined to cover a class of usage for assets. defines the core set of usage categories in detail. A statement of preference -- or usage preference -- is made about an asset. A statement of preference follows a simple data model where a preference is assigned to each of the categories of use in the vocabulary. A preference is either to allow or disallow the usage associated with the category. A statement of preference can indicate preferences about some, all, or none of the categories from the vocabulary. This can mean that no preference is stated for a given usage category. Some categories describe a proper subset of the usages of other categories. A preference that is stated for the more general category applies if no preference is stated for the more specific category. For example, the Automated Processing category might be assigned a preference that allows the associated usage. In the absence of any statement of preference regarding the AI Training category, that usage would be also be allowed, as AI Training is a subset of the Automated Processing category. In comparison, an explicit preference regarding AI Training might disallow that usage, while permitting other usage within the Automated Processing category. After processing a statement of preferences the recipient associates each category of use one of three preference values: "allowed", "disallowed", or "unknown". In the absence of a statement of preference, all usage categories are assigned a preference value of "unknown". The process for consulting a statement of preference is defined in . Different declaring parties might each make their own statement of preference regarding a particular asset. The process for managing multiple statements of preference is defined in . An exemplary syntax for statements of preference is defined in .

Conformance This document and describe how statements of preference are associated with assets. An implementation is conformant to these specifications if it correctly follows all normative requirements that apply to it. The process of obtaining a statement of preference has very limited scope for variation between implementations.

Applicability and Effect This specification provides a set of definitions for different categories of use, plus a system for associating simple preferences to each (allow, disallow, or no preference; see ). This specification does not provide any enforcement mechanism for those preferences, and conformance to it does not encompass whether preferences are actually respected during data processing. Preferences do not themselves create rights or prohibitions, either in the positive or the negative. Other mechanisms—technical, legal, contractual, or otherwise—might enforce stated preferences and thereby determine the consequences of following or not following a stated preference. An entity that receives usage preferences MAY choose to respect those preferences it has discovered, according to an understanding of how the asset is used, how that usage corresponds to the usage categories where preferences have been stated, and the applicable legal context. Usage preferences can be ignored due to express agreements between relevant parties, explicit provisions of law, or the exercise of discretion in situations where widely recognized priorities justify doing so. Priorities that could justify ignoring preferences include—but are not limited to—free expression, safety, education, scholarship, research, preservation, interoperability, and accessibility. The following lists examples of cases where other priorities could lead someone to ignore expressed preferences in a particular situation:

• People with accessibility needs, or organizations working on their behalf, might decide to ignore a preference disallowing Automated Processing () in order to access automated captions or generate accessible formats.
• A cultural heritage organization might decide to ignore a preference disallowing Automated Processing () in order to provide more useful, reliable, or discoverable access to historical web collections.
• An educational institution might decide to ignore a preference disallowing Foundation Model Production () in order to enable scholars to develop or use tools to facilitate scientific or other types of research.
• A website that permits user uploads might decide to ignore a preference disallowing Automated Processing () in order to develop or use tools that detect harmful content according to established terms of use.

Because enforcement is not provided by this specification, the consequences of ignoring preferences could vary depending upon how a given legal jurisdiction recognizes preferences.

Vocabulary Definition This section defines the categories of use in the vocabulary. shows the relationship between these categories:

Relationsehip Betweeen Categories of Use

Automated Processing Category The act of using automated processing on one or more assets to analyze text and data in order to generate information which includes but is not limited to patterns, trends and correlations. The use of assets for automated processing encompasses all the subsequent categories.

Foundation Model Production Category The act of using an asset to train or fine-tune a foundation model. Foundation models are large models that are produced using deep learning or other machine learning techniques. Foundation models are trained on very large numbers of assets so that they can be applied to a wide range of use cases. Foundation models typically possess generative capabilities in one or more media. Fine-tuning can specialize a general-purpose foundation model for a narrower set of use cases. The use of assets for Foundation Model Production is a proper subset of Automated Processing usage.

AI Output The act of using an asset in an AI-based system to generate outputs that are presented to clients of that system. This does not apply to any assets that are directly provided as inputs to the system by clients. This does not include the construction of the system, only its use. This includes the output of search results. This includes outputs that are presented to human users and outputs that are presented to automated clients. The AI Output category of use includes whatever model training is necessary to produce the models that are used in the generation of these outputs. The use of assets for AI Output is a proper subset of Automated Processing usage.

Search The Search category of use is a refinement of the AI Output category, with the addition of the following two conditions:

• A reference to the location that the asset was obtained is presented as part of the output.
• The asset can only be represented in the output with excerpts that are drawn verbatim from it.

If these conditions cannot be met, the asset then is not included in the output. With both these conditions, a preference to allow Search usage enables the presentation of links and titles in what is considered “traditional” search results. Like the AI Output category of use, Search includes whatever model training is necessary to produce the models that are used in the generation of these outputs. The use of assets for Search is a proper subset of AI Output usage.

Vocabulary Extensions Extensions to this vocabulary need to be defined in an RFC that updates this document. Any future extensions to this vocabulary MUST NOT introduce additional categories that include existing categories defined in the vocabulary. That is, new categories of use can be defined as a subset of an existing category, but not a superset. Systems that use this vocabulary might define their own extensions as part of a larger data model. describes how concepts from an alternative format might be mapped to this vocabulary.

Applying Statements of Preference After acquiring a statement of preference, which might use the process in , an application can determine the status of a specific usage category as follows:

1. If the statement of preference contains an explicit preference regarding that category of use -- either to allow or disallow -- that is the result.
2. Otherwise, if the usage category is a proper subset of another usage category, recursively apply this process to that category and use the result of that process.
3. Otherwise, no preference is stated.

This process results in one of three potential answers: allow, disallow, and unknown. Applications can use the answer to guide their behavior. One approach for dealing with an "unknown" outcome is to assign a default value. This document takes no position on what default might be assigned.

Combining Preferences The application might have multiple statements of preference, obtained using different methods or from different declaring parties. This might result in conflicting answers. Absent some other means of resolving conflicts, the following process applies to each usage category:

• If any statement of preference indicates that the usage is disallowed, the result is that the usage is disallowed.
• Otherwise, if any statement of preference allows the usage, the result is that the usage is allowed.
• Otherwise, no preference is stated.

This process ensures that the most restrictive preference applies.

More Specific Instructions A recipient of a statement of preferences that follows the model in might receive more specific instructions in two ways:

• Extensions to the vocabulary might define more specific categories of usage. Preferences about more specific categories override those of any more general category.
• Contractual agreements or other specific arrangements might override statements of preference.

For instance, a statement of preferences might indicate that the use of an asset is disallowed for AI Training. If arrangements, such as legal agreements, exist that explicitly permit the use of that asset, those arrangements likely apply despite the existence of machine-readable statements of preference, unless the terms of the arrangement explicitly say otherwise.

Exemplary Serialization Format This section defines an exemplary serialization format for preferences. The format describes how the abstract model could be turned into Unicode text or sequence of bytes. The format relies on the Dictionary type defined in . The dictionary keys correspond to usage categories and the dictionary values correspond to explicit preferences, which can be either y or n; see . For example, the following states a preference to allow automated processing (), disallow foundation model production (), and and states no preference for other categories other than subsets of these categories:

Usage Category Labels Each usage category in the vocabulary () is mapped to a short textual label. tabulates this mapping. Mappings for Categories

Category	Label	Reference
Automated Processing	bots
Foundation Model Production	train-ai
AI Output	ai-output
Search	search

These tokens are case sensitive. Tokens defined for a new usage category can only use lowercase latin characters (a-z), digits (0-9), "_", "-", ".", or "*". These are encoded using the mappings in .

Preference Labels The data model in used has two options for preferences associated with each category: allow and disallow. These are mapped to single byte Tokens () of y and n, respectively.

Text Encoding Structured Fields describes a byte-level encoding of information, not a text encoding. This makes this format suitable for inclusion in any protocol or format that carries bytes. Some formats are defined in terms of strings rather than bytes. These formats might need to decode the bytes of this format to obtain a string. As the syntax is limited to ASCII , an ASCII decoder or UTF-8 decoder can be used. This results in the strings that this document uses. Processing (see ) requires a sequence of bytes, so any format that uses strings needs to encode strings first. Again, this process can use ASCII or UTF-8.

Syntax Extensions There are two ways by which this syntax might be extended: the addition of new labels and the addition of parameters. New labels might be defined to correspond to new usage categories. addresses the considerations for defining new categories. New labels might also be defined for other types of extension that do not assign a preference to a usage category. In either case, when processing a parsed Dictionary to obtain preferences, any unknown labels MUST be ignored. The Dictionary syntax () can associate parameters with each key-value pair. This document does not define any semantics for any parameters that might be included. When processing a parsed Dictionary to obtain preferences, any unknown parameters MUST be ignored. In either case, new extensions need to be defined in an RFC that updates this document.

Processing Algorithm To process a series of bytes to recover the stated preferences, those bytes are parsed into a Dictionary (), then preferences are assigned to each usage category in the vocabulary. This algorithm produces a keyed collection of values, where each key has at most one value and optional parameters. To obtain preferences, iterate through the defined categories in the vocabulary. For the label that corresponds to that category (see ), obtain the corresponding value from the collection, disregarding any parameters. A preference is assigned as follows:

• If the value is a Token with a value of y, the associated preference is to allow that category of use.
• If the value is a Token with a value of n, the associated preference is to disallow that category of use.
• Otherwise, no preference is stated for that category of use.

Note that this last alternative includes the key being absent from the collection, values that are not Tokens, and Token values that are other than y or n. All of these are not errors, they only result in no preference being inferred. It is important to note that if the same key appears multiple times, only the last value is taken. This means that duplicating a key could result in unexpected outcomes. For example, the following expresses no preferences: If the parsing of the Dictionary fails, no preferences are stated. This includes where keys include uppercase characters, as this format is case sensitive (more correctly, it operates on bytes, not strings). This document does not define a use for parameters. Where parameters are used, only those parameters associated with the value that is selected according to . Parameters can therefore be carried for any preference value, including where no preference is expressed. For example, the following train-ai preference has parameters even though no preference is expressed: This process produces an abstract data model that assigns a preference to each usage category as described in .

Alternative Formats This format is only an exemplary way to represent preferences. The data model described in , can be used without this serialization. Any alternative format needs to define the mapping both from that format to the model used in this document and from the model to the alternative format. This includes any potential for extensions (). The mapping between the data model and the alternative format does not need to be complete, it only needs to be clear and unambiguous. For example, an alternative format might only provide the ability to convey preferences for a subset of the categories of use. A mapping might then define that no preference is associated with other categories.

Security Considerations Preferences are not a security mechanism. addresses what it means to express a preference. Processing a concrete instantiation of the exemplary format described in is subject to the security considerations in .

IANA Considerations This document has no IANA actions.

References Normative References This document describes a set of data types and associated algorithms that are intended to make it easier and safer to define and handle HTTP header and trailer fields, known as "Structured Fields", "Structured Headers", or "Structured Trailers". It is intended for use by specifications of new HTTP fields. This document obsoletes RFC 8941. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References ISO/IEC 10646-1 defines a large character set called the Universal Character Set (UCS) which encompasses most of the world's writing systems. The originally proposed encodings of the UCS, however, were not compatible with many current applications and protocols, and this has led to the development of UTF-8, the object of this memo. UTF-8 has the characteristic of preserving the full US-ASCII range, providing compatibility with file systems, parsers and other software that rely on US-ASCII values but are transparent to other values. This memo obsoletes and replaces RFC 2279. Google Mozilla

Acknowledgments The following individuals made significant contributions to this document: