]> Common Crawl Foundation

thom@commoncrawl.org

AI Preferences AI Preferences Opt-Out Vocabulary This document proposes a vocabulary for expressing content preferences for rightsholders who wish to manage the use of their content in AI training. This vocabulary allows publishers to express preferences through metadata or content-delivery protocols. The vocabulary can be applied at different levels of granularity and incorporates preferences for permissions, usage scope, and data retention, providing a foundation for interoperability across various Internet protocols. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the AI Preferences mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction As AI models become more reliant on large-scale data (driven by scaling laws that link model performance to dataset size), content publishers seek ways to control how their content is used in training these models. This draft provides a vocabulary that enables publishers to signal preferences for AI training concerning their content.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Scope The AI-PREF vocabulary is limited to expressing content preferences for AI training and does not include enforcement mechanisms or client authentication. Default opt-in or opt-out statuses are beyond the scope of this proposal, as it focuses solely on establishing a standard for signalling explicit preferences. In cases where no preferences are signalled, the decision on whether this constitutes an opt-in or opt-out should be determined at the policy level downstream. It is important to note that preference signals are advisory.

Vocabulary Elements / Preference Signals

Permission Basic indicators of whether content can be used for AI training.

• allow_training: Boolean
• restricted_training: public, non-commercial, internal, licensed

Purpose Defines acceptable uses in training.

• purpose: String:
  • generation: Creating models that are capable of generating content
  • embedding: Converting content to vector representations
    • classification: Categorising or labelling content
    • summary: Creating condensed versions of content
    • paraphrase: Creating derivative versions of content
    • quotation: Repetition of a passage or fragment of original content
    • translation: Converting content between languages

Temporal Restrictions Specifies the date range for training use.

• effective_date: ISO 8601 Date string
• expiration_date: ISO 8601 Date string

Content-Specific Granularity Defines the scope of applicability. Refers to the level at which preferences apply within the content.

• scope: global, content-specific, conditional

Content Type Specifies content types the preference applies to.

• mime_type: text, image, video, audio, application, .

Derivative Content Allows or restricts derivatives like summaries.

• allow_derivatives: Boolean
• derivative_type: summary, paraphrase, translation

Data Retention Defines content retention period post-training.

• retention_period: ISO 8601 Duration string (e.g., P3Y6M4DT12H30M5S)

Preference Persistence Indicates if preferences should persist in derived datasets, or be optional. A derived dataset is the result of processing, transforming, or extracting information from the original source, such as aggregated statistics and summaries, or subsets of data.

• metadata_persistence: Boolean

Precedence Conflicts should be resolved by assigning precedence values (e.g., high, medium, low) to rules, with a defined hierarchy that allows content producers to override publishers, domain operators, and others as necessary.

• precedence: Sets priority when preferences conflict with other layered preferences.

Geographic Restrictions Specifies regions where preferences apply, ISO 3166-1.

• geo_limitations: Specifies geographic regions where training permissions apply.

Implementation Considerations Implementing the AI-PREF vocabulary effectively can be accomplished using various mechanisms, depending on the needs and existing infrastructure of content publishers. Approaches include, but are not limited to, using HTTP headers, possible extensions to (), and (for example) <meta> tags and other embedded data (such as EXIF) for sub-document-level control.

HTTP Headers Publishers can use HTTP headers to communicate AI-PREF preferences directly in response to client requests. This approach allows fine-grained control and easy integration into existing server configurations. Example header: This header specifies that the content can be used for text generation and classification, with a retention period of 3 years, 6 months, 4 days, 12 hours, 30 minutes, and 5 seconds. The syntax and options should be carefully chosen to ensure compatibility with common web servers and clients.

Robots Exclusion Protocol (REP) For publishers who already use REP (as defined in RFC9309), extending REP rules to include AI-PREF preferences could be beneficial. Example rule: This REP rule specifies that all user agents are allowed to use the content for non-commercial AI training, limited to embedding and summarisation purposes. Further extensions to REP could specify additional constraints, such as geographic limitations or temporal restrictions.

<meta> Tags for Sub-Document Level Control To specify AI-PREF preferences at the level of individual HTML documents or specific parts of a document, <meta> tags and HTML attributes can be used. Example <meta> tag: ]]> Example HTML attribute: ]]> The methods above specify that AI training is not allowed for the content of this document, with no retention period permitted. <meta> tags can be used to provide specific content preferences for a specific piece of content, and thus provide a flexible way to manage AI training signals at a more granular level.

“Well-Known” Locations According to , “well-known” locations can serve metadata or configuration information that is easily discoverable by automated clients. AI-PREF preferences can be published at a “well-known” URL. There is already the Text and Data Mining Reservation Protocol (TDMRep) which has the same or overlapping intent. Example: At this URL, a JSON or other structured format can specify AI-PREF preferences for the entire domain or specific content types. Example JSON 1: Example JSON 2: This approach simplifies discovery for automated clients and provides a centralised way to communicate content preferences across a domain. TDMRep Example: A rightsholder could expose a “well-known” TDMRep file at: https://example.com/.well-known/tdmrep Example TDMRep JSON Content:

Embedded Metadata Preferences for multimodal data can be embedded directly into file metadata (such as EXIF or XMP) as self-contained control signals. Compatibility and tamper resistance (e.g. signing) should be considered. Example EXIF: Example PDF Metadata Using XMP: Text mining allowed; Data sharing restricted ]]> Preferences can be applied at the file level, or even to specific components (e.g., chapters in a PDF or frames in a video). Example WEBVTT: 00:01:00.000 Usage Preferences: allow_training=true; purpose=generation,classification 00:01:01.000 --> 00:05:00.000 Usage Preferences: allow_training=false; ]]>

Content Credentials (ISO 22144) TBD

ISCC (ISO 24138) TBD

Example Usage Scenarios TODO examples

Security Considerations This document does not affect the security of the Internet. AI-PREF preferences do not include enforcement mechanisms, which should be addressed by AI model developers. Publishers should be aware that preferences may not prevent unauthorised use and may rely on mutual agreements or legal protections.

IANA Considerations This document does not require any immediate IANA actions but may suggest future registry entries for the vocabulary terms to support interoperability.

References Normative References In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. This second document defines the general structure of the MIME media typing system and defines an initial set of media types. [STANDARDS-TRACK] This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching. This memo defines a path prefix for "well-known locations", "/.well-known/", in selected Uniform Resource Identifier (URI) schemes. In doing so, it obsoletes RFC 5785 and updates the URI schemes defined in RFC 7230 to reserve that space. It also updates RFC 7595 to track URI schemes that support well-known URIs in their registry. Informative References Google LLC. The Robots Exclusion Protocol defined in [RFC9309] specifies the user-agent rule for targeting automatic clients either by prefix matching their self-defined product token or by a global rule * that matches all clients. This document extends [RFC9309] by defining a new rule for targeting automatic clients based on the clients' purpose for accessing the service.

Table of Preference Signals This table defines terms and values that specify metadata preferences for the use of content in AI training. Each term includes a description of its purpose and example values:

Term	Values	Description	Example
allow_training	Boolean	Basic indicator of whether content can be used for AI training	allow_training: false
purpose	String: generation, classification, summarisation, embedding, etc	Defines acceptable applications for training e.g. fine-tuning, classification, summarisation, etc	purpose: classification, summarisation
effective_date	Date string, ISO 8601	Start date of when permissions take effect	effective_date: 2024-10-30T15:52:55.440238
expiration_date	Date string, ISO 8601	Date after which permissions no longer apply	expiration_date: 2024-10-30T15:52:55.440238
scope	String: global, content-specific, conditional	Defines whether the preferences apply universally, to specific content, or under certain conditions	scope: content-specific
mime_type	text, image, video, audio	Specifies the type(s) of content the preference applies to	mime_type: text, image
allow_derivatives	Boolean	Indicates whether derivative works (summaries, paraphrasing) are allowed based on content	allow_derivatives: true
derivative_type	String: summary, paraphrase, translation	Lists permissible types is allow_derivatives is true	derivative_type: summary, paraphrase
retention_period	Duration string, ISO 8601	Specifies how long content may be retained after use (e.g. after training).	P3Y6M4DT12H30M5S representing three years, six months, four days, twelve hours, thirty minutes, and five seconds.
preference_persistence	Boolean	Whether preferences must persist with derived data, boolean for either required or optional	preference_persistence: true
precedence	String:high, medium, low	Sets priority when preferences conflict with other layered preferences	precedence: high
geo_limitations	Location codes, ISO 3166	Specifies geographic regions where training permissions apply	geo_limitations: EU, US

Acknowledgments

• Greg Lindahl
• Sebastian Nagel
• Gary Illyes
• Mark Nottingham
• Suresh Krishnan
• Martin Thomson
• Paul Keller
• Leonard Rosenthol
• Special thanks to the program committee and contributing members of the IAB AI-CONTROL Workshop, and aipref Working Group.