]> Cloudflare

maxime.guerreiro@gmail.com

Cloudflare

ot-ietf@thibault.uk

Web and Internet Transport Web Bot Auth not-yet This document describes a JSON based format for clients using to advertise information about themselves. This document describes a JSON-based "Signature Agent Card" format for signature agent using to advertise metadata about themselve. This includes identity, purpose, rate expectations, and cryptographic keys. It also establishes an IANA registry for Signature Agent Card parameters, enabling extensible and interoperable discovery of agent information. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the Web Bot Auth Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction Signature Agents are entities that originate or forward signed HTTP requests on behalf of users or services. They include bots developers, platforms providers, and other intermediaries using . These agents often need to identify themselves, and establish trust with origin servers. Today, the mechanisms for doing so are inconsistent: some rely on User-Agent strings (e.g. MyCompanyBot/1.0), others on IP address lists hosted on file servers (e.g. badbots.com), and still others on out-of-band definitions (e.g. documentation on docs.example.com/mybot). This diversity makes it difficult for operators and origin servers to reliably discover and share a Signature Agent’s purpose, contact information, or rate expectations. Existing discovery mechanisms, such as , do not have the necessary granularity, and pursue different goals. This document introduces a JSON-based "Signature Agent Card" format for Signature Agents, to be published in registries and discovered by servers. It also creates a new IANA registry of "Signature Agent Card Parameters" to ensure extensibility and consistency of future attributes.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Signature Agent Card Signature-Agent header is defined in . This section describes Signature Agent Card, a JSON object containing parameters describing the Signature Agent. Unless otherwise specified, all parameters in this document are OPTIONAL. Unknown parameters MUST be ignored. All string values are UTF-8.

Name The name parameter provides a friendly identifier for the Signature Agent. Example

• ExampleBot
• My remote browser company

Contact The contact parameter provides an email address or reliable communication channel. Example * bot-support@example.com * https://example.com/contact

Logo The logo parameter provides an image reference for visual identification. Example

• data:image/svg+xml;base64,deadbeef
• https://example.com/logo.png

Expected user agent The expected-user-agent parameter specifies one or more User-Agent strings as defined in or prefix matches. Prefixes MAY use * as a wildcard. Example

• Mozilla/5.0 ExampleBot

robots.txt product token The rfc9309-product-token parameter specifies the product token used for robots.txt directives per . Example

• ExampleBot

robots.txt compliance The rfc9309-compliance parameter lists directives from robots.txt that the agent implements. Example

• ["User-Agent", "Disallow"]
• ["User-Agent", "Disallow", "CrawlDelay"]

Trigger The trigger parameter indicates the operational mode of the agent. Valid values:

1. "fetcher" - request initiated by the user
2. "crawler" - autonomous scanning

Purpose The purpose parameter describes the intended use of collected data. Values SHOULD be drawn from a controlled vocabulary, such as . Example

• search
• tdm

Targeted content The targeted-content parameter specifies the type of data the agent seeks. Example

• SEO analysis
• Vulnerability scanning
• Ads verification

Rate control The rate-control parameter indicates how origins can influence the agent’s request rate. TODO: specify a format Example

• CrawlDelay in robots.txt (non-standard)
• Custom tool
• 429 +

Rate expectation The rate-expectation parameter specifies anticipated request volume or burstiness. TODO: consider a format such as avg=10rps;max=100rps Example

• 500 rps
• Spikes during reindexing

Known URLs The known-urls parameter lists predictable endpoints accessed by the agent. Example

• /
• /ads.txt
• /favicon.ico
• /index.html

Keys The keys parameter contains a JWKS as defined in . If keys is present, it is RECOMMENDED that the card is signed using . Content-Digest header MUST be included in the covered components. TODO: describe signature, CWS keys. Example

• https://example.com/.well-known/http-message-signatures-directory
• JWKS-directory

Discovery Discovery is done via a registry. A registry is a list of URLs, each one pointing to a signature card. URLS can be:

• https
• http
• data

Example

Out-of-band communication between client and origin A service submitting their key to an origin, or the origin manually adding a service to their trusted list.

Public list Could be a GitHub repository like the public suffix list. The issue is the gating of such repositories, and therefore its governance.

Signature-Agent-Card header This allows for backward compatibility with existing header agent filtering, and an upgrade to a cryptographically secured protocol.

Security Considerations Malicious actors may put properties which are not theirs in the registry. Client SHOULD verify signature if they are present.

Privacy Considerations TODO

IANA Considerations

Signature Agent Card Parameters Registry IANA is requested to create a new "Signature Agent Card Parameters" registry in a new "Signature Agent Card" page to list metadata for signature agent protocols.

Registration template New registrations need to list the following attributes:

Parameter Name:

The name requested (e.g. "useragent"). This name is case sensitive. Names may not match other registered names in a case-insensitive manner unless the Designated Experts state that there is a compelling reason to allow an exception

Parameter Description:

Brief description of the Header Parameter

Change Controller:

For Standards Track RFCs, list the "IESG". For others, give the name of the responsible party. Other details (e.g., postal address, email address, home page URI) may also be included.

Reference:

Where this parameter is defined

Notes:

Any notes associated with the entry

New entries in this registry are subject to the Specification Required registration policy (). Designated experts need to ensure that the token type is defined to be used for both token issuance and redemption. Additionally, the experts can reject registrations on the basis that they do not meet the security and privacy requirements defined in TODO.

Initial Registry content This section registers the Signature Agent Card Parameter names defined in in this registry.

Name Parameter

Parameter Name:

name

Parameter Description:

A friendly name for your signature agent.

Change Controller:

IETF

Reference:

Notes:

N/A

Contact Parameter

Parameter Name:

contact

Parameter Description:

Email or any other reliable communication channel

Change Controller:

IETF

Reference:

Notes:

N/A

Logo Parameter

Parameter Name:

logo

Parameter Description:

Image for a quick visual identification

Change Controller:

IETF

Reference:

Notes:

N/A

Expected User Agent Parameter

Parameter Name:

expected-user-agent

Parameter Description:

String or fragment patterns

Change Controller:

IETF

Reference:

Notes:

N/A

RFC9309 Product Token Parameter

Parameter Name:

rfc9309-product-token

Parameter Description:

Robots.txt product token your signature-agent satisfies.

Change Controller:

IETF

Reference:

Notes:

N/A

RFC9309 Compliance Parameter

Parameter Name:

rfc9309-compliance

Parameter Description:

Does your signature-agent respect robots.txt.

Change Controller:

IETF

Reference:

Notes:

N/A

Trigger Parameter

Parameter Name:

trigger

Parameter Description:

Fetcher/Crawler

Change Controller:

IETF

Reference:

Notes:

N/A

Purpose Parameter

Parameter Name:

purpose

Parameter Description:

Intended use for the collected data

Change Controller:

IETF

Reference:

Notes:

N/A

Targeted Content Parameter

Parameter Name:

targeted-content

Parameter Description:

Type of data your agent seeks

Change Controller:

IETF

Reference:

Notes:

N/A

Rate control Parameter

Parameter Name:

rate-control

Parameter Description:

How can an origin control your crawl rate

Change Controller:

IETF

Reference:

Notes:

N/A

Rate expectation Parameter

Parameter Name:

rate-expectation

Parameter Description:

Expected traffic and intensity

Change Controller:

IETF

Reference:

Notes:

N/A

Known URLs Parameter

Parameter Name:

known-urls

Parameter Description:

Predictable endpoint accessed

Change Controller:

IETF

Reference:

Notes:

N/A

Keys Parameter

Parameter Name:

keys

Parameter Description:

JWKS Endpoint

Change Controller:

IETF

Reference:

Notes:

N/A

References Normative References Open Future Mozilla This document defines a vocabulary for expressing preferences regarding how digital assets are used by automated processing systems. This vocabulary allows for the declaration of restrictions or permissions for use of digital assets by such systems. Cloudflare This document describes a method for clients using [HTTP-MESSAGE-SIGNATURES] to advertise their signing keys. It defines a key directory format based on JWKS as defined in Section 5 of [JWK], as well as new HTTP Method Context to allow for in-band key discovery. This document describes a mechanism for creating, encoding, and verifying digital signatures or message authentication codes over components of an HTTP message. This mechanism supports use cases where the full HTTP message may not be known to the signer and where the message may be transformed (e.g., by intermediaries) before reaching the verifier. This document also describes a means for requesting that a signature be applied to a subsequent HTTP message in an ongoing HTTP exchange. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared by all versions. In this definition are core protocol elements, extensibility mechanisms, and the "http" and "https" Uniform Resource Identifier (URI) schemes. This document updates RFC 3864 and obsoletes RFCs 2818, 7231, 7232, 7233, 7235, 7538, 7615, 7694, and portions of 7230. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document defines HTTP caches and the associated header fields that control cache behavior or indicate cacheable response messages. This document obsoletes RFC 7234. This document specifies additional HyperText Transfer Protocol (HTTP) status codes for a variety of common situations. [STANDARDS-TRACK] A JSON Web Key (JWK) is a JavaScript Object Notation (JSON) data structure that represents a cryptographic key. This specification also defines a JWK Set JSON data structure that represents a set of JWKs. Cryptographic algorithms and identifiers for use with this specification are described in the separate JSON Web Algorithms (JWA) specification and IANA registries established by that specification. This document defines how to use the Diffie-Hellman algorithms "X25519" and "X448" as well as the signature algorithms "Ed25519" and "Ed448" from the IRTF CFRG elliptic curves work in JSON Object Signing and Encryption (JOSE). This specification defines a method for computing a hash value over a JSON Web Key (JWK). It defines which fields in a JWK are used in the hash computation, the method of creating a canonical form for those fields, and how to convert the resulting Unicode string into a byte sequence to be hashed. The resulting hash value can be used for identifying or selecting the key represented by the JWK that is the subject of the thumbprint. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Many protocols make use of points of extensibility that use constants to identify various protocol parameters. To ensure that the values in these fields do not have conflicting uses and to promote interoperability, their allocations are often coordinated by a central record keeper. For IETF protocols, that role is filled by the Internet Assigned Numbers Authority (IANA). To make assignments in a given registry prudently, guidance describing the conditions under which new values should be assigned, as well as when and how modifications to existing values can be made, is needed. This document defines a framework for the documentation of these guidelines by specification authors, in order to assure that the provided guidance for the IANA Considerations is clear and addresses the various issues that are likely in the operation of a registry. This is the third edition of this document; it obsoletes RFC 5226. Informative References This specification describes how to use bearer tokens in HTTP requests to access OAuth 2.0 protected resources. Any party in possession of a bearer token (a "bearer") can use it to get access to the associated resources (without demonstrating possession of a cryptographic key). To prevent misuse, bearer tokens need to be protected from disclosure in storage and in transport. [STANDARDS-TRACK] n.d. Team Digitale, Italian Government Red Hat Microsoft This document defines the RateLimit-Policy and RateLimit HTTP header fields for servers to advertise their quota policies and the current service limits, thereby allowing clients to avoid being throttled. This document specifies version 1.3 of the Transport Layer Security (TLS) protocol. TLS allows client/server applications to communicate over the Internet in a way that is designed to prevent eavesdropping, tampering, and message forgery. This document updates RFCs 5705 and 6066, and obsoletes RFCs 5077, 5246, and 6961. This document also specifies new requirements for TLS 1.2 implementations. This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching.

Test Vectors TODO

Implementations TODO

Acknowledgments TODO The editor would also like to thank the following individuals (listed in alphabetical order) for feedback, insight, and implementation of this document -

Changelog v00

• Initial draft