]> Cloudflare

ot-ietf@thibault.uk

Web and Internet Transport Web Bot Auth not-yet This document describes an architecture for identifying automated traffic using . The goal is to allow automated HTTP clients to cryptographically sign outbound requests, allowing HTTP servers to verify their identity with confidence. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the Web Bot Auth Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction Agents are increasingly used in business and user workflows, including AI assistants, search indexing, content aggregation, and automated testing. These agents need to reliably identify themselves to origins for several reasons:

1. Regulatory compliance requiring transparency of automated systems
2. Origin resource management and access control
3. Protection against impersonation and reputation management
4. Service level differentiation between human and automated traffic

Current identification methods such as IP allowlisting, User-Agent strings, or shared API keys have significant limitations in security, scalability, and manageability. This document defines an architecture enabling agents to cryptographically identify themselves using . It proposes that every request from bots to be signed by a private key owned by its provider. This way, every origin can validate the service identity.

Motivation There is an increase in agent traffic on the Internet. Many agents choose to identify their traffic today via IP Address lists and/or unique User-Agents. This is often done to demonstrate trust and safety claims, support allowlisting/denylisting the traffic in a granular manor, and enable sites to monitor and rate limit per agent operator. However, these mechanisms have drawbacks:

1. User-Agent, when used alone, can be spoofed meaning anyone may attempt to act as that agent. It is also overloaded - an agent may be using Chromium and wish to present itself as such to ensure rendering works, yet it still wants to differentiate its traffic to the site.
2. IP blocks alone can present a confusing story. IPs on cloud plaforms have layers of ownership - the platform owns the IP and registers it in their published IP blocks, only to be re-published by the agent with little to bind the publication to the actual service provider that may be renting infra. Purchasing dedicated IP blocks is expensive, time consuming, and requires significant specialist knowledge to set up. These IP blocks may have prior reputation history that needs to be carefully inspected and managed before purchase and use.
3. An agent may go to every website on the Internet and share a secret with them like a Bearer from . This is impractical to scale for any agent beyond select partnerships, and insecure, as key rotation is challenging and becomes less secure as the consumers scale.

Using well-established cryptography, we can instead define a simple and secure mechanism that empowers small and large agents to share their identity.

HTTP layer choice This architecture operates solely at the HTTP layer. It allows signatures to be generated and verified without modifying the transport layer or TLS stack. It enables flexible deployment across proxies, gateways, and origin servers, and aligns with existing tooling and infrastructure that already inspect and manipulate HTTP headers. Because the signature is embedded in the request itself, it travels with the message through intermediaries, preserving end-to-end verifiability even when requests are forwarded or transformed within the HTTP layer.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. The following terms are used throughout this document:

User

An entity initiating requests through an agent. May be a human operator or another system.

Agent

An orchestrated user agent (e.g. Chromium, CURL). It implements the HTTP protocol and constructs valid HTTP requests with signatures.

Origin

An HTTP server receiving signed requests that implements the HTTP protocol and verifies signatures. It acts as a verifier of the signature as defined by .

Architecture | | | | | | material | | | User +--- Request -->| Agent | | Origin | | | | +--- Request + Signature -->| | | | | |<-------- Response --------+ | | |<-- Response --+ | | | | | | | | | +--------+ +---------+ +----------+ ]]> A User initiates an action requiring the Agent to perform an HTTP request. The Agent constructs the request, generates a signature using its signing key, and includes it in the request as defined in along with the Signature-Agent header for discovery for its verification key. Upon receiving the request, the Origin ensures it has the verification key for the Agent, validates the signature, and processes the request if the signature is valid.

Deployment Models Signature verification can be performed either directly by origins or delegated to a fronting proxy. Direct verification by origins provides simplicity and control. Proxy verification offloads processing and enables shared caching across multiple origins. The choice depends on traffic volume and operational requirements.

Generating HTTP Message Signature defines components to be signed. Agents MUST include at least one of the following components:

@authority

as defined in

@target-uri

as defined in

Agents MUST include the following @signature-params as defined in

created

as defined in

expires

as defined in

keyid

MUST be a base64url JWK SHA-256 Thumbprint as defined in for RSA and EC, and in for ed25519.

tag

MUST be web-bot-auth

The signing key is available to the agent at request time. Algorithms should be registered with IANA as part of HTTP Message Signatures Algorithm registry. The creation of the signature is defined in . It is RECOMMENDED the expiry to be no more than 24 hours.

Signature-Agent Signature-Agent is an HTTP Method context header defined in Section 4.1 of . It is RECOMMENDED that the Agent sends requests with Signature-Agent header, as described in . If the header is to be sent, one of its members MUST be signed as a component as defined in . This results in the following components to be signed It is RECOMMENDED that the key matches the signature label.

Anti-replay Origins MAY want to prevent signatures from being spoofed or used multiple times by bad actors and thus require a nonce to be added to the @signature-params. This is described in . Agents SHOULD extend @signature-parameters defined in as follows:

nonce

base64url encoded random byte array. It is RECOMMENDED to use a 64-byte array.

Client MUST ensure that this nonce is unique for the validity window of the signature, as defined by created and expires attributes.

Additional headers Agents MAY include additional components, such as specific HTTP headers, in the signature. This can be prompted by the origin requesting additional headers, as described in , or initiated by the agent to provide more information within the signature scope. For example, an agent might include an HTTP header expressing its intent and sign it. Origins MAY ignore certain headers at their own discretion, and request a new signature, as described in .

Sending a request An Agent SHOULD send a request with the signature generated above. Updating the architecture diagram, the flow looks as follow. | | | | material | | | Agent | | Origin | | | .-------------------------------------------------------------------. | | | +-----| GET /path/to/resource |------->| | | | | Signature: abc== | | | +---------+ | Signature-Input: sig=("@authority" "signature-agent";key="sig");\ | +----------+ | created=1700000000;\ | | expires=1700011111;\ | | keyid="ba3e64==";\ | | tag="web-bot-auth" | | Signature-Agent: sig="https://signer.example.com" | '-------------------------------------------------------------------' ]]> The Agent SHOULD send requests with two headers

1. Signature defined in
2. Signature-Input defined in

Mentioned in , the Agent MAY send requests with Signature-Agent header.

Requesting a Message signature defines the Accept-Signature field which can be used to request a Message Signature from a client by an origin. An Origin MAY choose to request signatures from clients that did not initially provide them. If requesting, Origins MUST use the same parameters as those defined by the . The status code SHOULD be 403 Forbidden as defined in . Origin MAY request a new signature with tag "web-bot-auth" even if a nonce is provided, for example if it believes the nonce is a replay, or if it doesn't store nonces and thus requests new signatures every time. The status code SHOULD be 429 Too Many Requests as defined in .

Validating Message signature Upon receiving an HTTP request, the origin has to verify the signature. The algorithm is provided in . Similar to a regular User-Agent check, this happens at the HTTP layer, once headers are received. Additional requirements are placed on this validation:

• During step 1 to 3 included, if the Origin fails to parse the provided Signature, Signature-Input, or Signature-Agent headers, it MAY respond with status code 400 Bad Request as defined in .
• During step 4, the Origin MAY discard signatures for which the tag is not set to web-bot-auth.
• During step 5, the Origin MAY discard signatures for which they do not know the keyid.
• During step 5, if the keyid is unknown to the origin, they MAY fetch the provider directory as indicated by Signature-Agent header defined in Section 4 of .

Origin MAY require the nonce to satisfy certain constraint: be globally unique using a global nonce store, be unique to a specific location or time window using a local cache, or not constraint at all.

Key Distribution and Discovery This section describes the discovery mechanism for the agent directory. The reference for discovery is a FQDN. It SHOULD provide a directory hosted on the well known registered in Section 4 of . Example:

Out-of-band communication between client and origin A service submitting their key to an origin, or the origin manually adding a service to their trusted list.

Public list Could be a GitHub repository like the public suffix list. The issue is the gating of such repositories, and therefore its governance.

Signature-Agent header This allows for backward compatibility with existing header agent filtering, and an upgrade to a cryptographically secured protocol. See for more details.

Security Considerations

Use of TLS We reassess . Clients SHOULD use TLS (https) or equivalent transport security when making requests with Message signatures. Failing to do so exposes the Message signature to numerous attacks that could give attackers unintended access. This include reverse proxy and their consideration presented in . An origin SHOULD refuse Signature headers when communicated over an unsecured channel.

Performance Impact Origins should account for the overhead of signature verification in their operations. A local cache of public keys reduces network requests and verification latency. The choice of signing algorithm impacts CPU requirements. Origins should monitor verification latency and set appropriate timeouts to maintain service levels under load.

Nonce validation Clients are the one controlling the nonce. While mandates that clients MUST provide a globally unique nonce, it is the origin's responsibility to enforce it. Different validation policies have different performance and operational considerations. Global uniqueness requires a global nonce store. Some origins may find that their use case can tolerate sharding on location, timing, or other properties.

Key Compromise Response An agent signing key might get compromised. If that happens, the agent SHOULD cease using the compromised key as soon as possible, notify affected origins if possible, and generate a new key pair. To minimise the impact of a key compromise, the origin should support rapid key rotation and monitor for suspicious signature patterns.

Shared Secrets Considered Harmful Implementations MUST NOT use shared HMAC defined in . Shared secrets break non-repudiation and make auditing difficult. Each automated client SHOULD use a unique asymmetric keypair to ensure attribution, support key rotation, and enable effective revocation if needed.

Key Reuse Considered Harmful Implementations SHOULD NOT reuse a signing key for different purposes. For example, if an agent implementor has two agents they want to differentiate, these should use distinct signing keys and signing key directories.

Reverse proxy consideration An origin may be placed behind a reverse proxy, which means the proxy will see the Signature and Signature-Agent headers before the origin does. A proxy SHOULD NOT strip the Signature or Signature-Agent headers from requests. A proxy SHOULD NOT replay signatures against other reverse proxies used by the origin, as this allows impersonation of the principal signature agent. Origins MAY require a specific nonce policy to prevent such malicious behaviour and decide to validate the signature themselves. This has to be done in accordance with . For example, an origin could require a nonce derived from public information (such as the current date), mandate nonce chaining (where each nonce is the hash of the previous one), or provide its own nonce in an Accept-Signature response to challenge the agent. Such policies MAY incur additional round-trip between the client and the origin to convey accept-signature header, or deployment specific exchanges.

Signature-Agent labeling An intermediary is allowed to relabel an existing signature when processing the message, per . This MAY apply to Signature-Agent, when included in the request as defined in , An intermediary updating the member key MUST update the components of the associated signatures accordingly. For instance, an intermediary updating the Signature-Agent from sig2 to sig3 on the example provided in would result in the following Signature, Signature-Input, and Signature-Agent header. Signature is unchanged as the base is similar. Both Signature-Agent and Signature-Input reflect the update from sig2 to sig3. . # Privacy Considerations

Public Identity This architecture assumes that automated clients identify themselves explicitly using digital signatures. The identity associated with a signing key is expected to be publicly discoverable for verification purposes. This reduces anonymity and allows receivers to associate requests with specific agents. If an agent wishes not to identify itself, this is not the right choice of protocol for it.

No Human Correlation The key used for signing MUST NOT be tied to a specific human individual. Keys SHOULD represent a role, company, or automation identity (e.g., "news-aggregator- bot", "example-crawler-v1"). This avoids accidental exposure of personally identifiable information and prevents the misuse of keys for user tracking or profiling.

Minimizing Tracking Risks To limit tracking risks, implementations SHOULD avoid long-lived, globally unique key identifiers unless strictly necessary. Key rotation SHOULD be supported, and clients SHOULD take care to avoid signing information that could be used to correlate activity across contexts, especially where sensitive user data is involved.

IANA Considerations This document has no IANA actions.

References Normative References Cloudflare This document describes a method for clients using [HTTP-MESSAGE-SIGNATURES] to advertise their signing keys. It defines a key directory format based on JWKS as defined in Section 5 of [JWK], as well as new HTTP Method Context to allow for in-band key discovery. This document describes a mechanism for creating, encoding, and verifying digital signatures or message authentication codes over components of an HTTP message. This mechanism supports use cases where the full HTTP message may not be known to the signer and where the message may be transformed (e.g., by intermediaries) before reaching the verifier. This document also describes a means for requesting that a signature be applied to a subsequent HTTP message in an ongoing HTTP exchange. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared by all versions. In this definition are core protocol elements, extensibility mechanisms, and the "http" and "https" Uniform Resource Identifier (URI) schemes. This document updates RFC 3864 and obsoletes RFCs 2818, 7231, 7232, 7233, 7235, 7538, 7615, 7694, and portions of 7230. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document defines HTTP caches and the associated header fields that control cache behavior or indicate cacheable response messages. This document obsoletes RFC 7234. This document specifies additional HyperText Transfer Protocol (HTTP) status codes for a variety of common situations. [STANDARDS-TRACK] This document defines how to use the Diffie-Hellman algorithms "X25519" and "X448" as well as the signature algorithms "Ed25519" and "Ed448" from the IRTF CFRG elliptic curves work in JSON Object Signing and Encryption (JOSE). This specification defines a method for computing a hash value over a JSON Web Key (JWK). It defines which fields in a JWK are used in the hash computation, the method of creating a canonical form for those fields, and how to convert the resulting Unicode string into a byte sequence to be hashed. The resulting hash value can be used for identifying or selecting the key represented by the JWK that is the subject of the thumbprint. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References This specification describes how to use bearer tokens in HTTP requests to access OAuth 2.0 protected resources. Any party in possession of a bearer token (a "bearer") can use it to get access to the associated resources (without demonstrating possession of a cryptographic key). To prevent misuse, bearer tokens need to be protected from disclosure in storage and in transport. [STANDARDS-TRACK] This document specifies version 1.3 of the Transport Layer Security (TLS) protocol. TLS allows client/server applications to communicate over the Internet in a way that is designed to prevent eavesdropping, tampering, and message forgery. This document updates RFCs 5705 and 6066, and obsoletes RFCs 5077, 5246, and 6961. This document also specifies new requirements for TLS 1.2 implementations.

Test Vectors

RSASSA-PSS Using SHA-512 The test vectors in this section use the RSA-PSS key defined in . This section includes non-normative test vectors that may be used as test cases to validate implementation correctness.

Signature-Agent absent from the request This example presents a minimal signature using the rsa-pss-sha512 algorithm over test-request. The request does not contain a Signature-Agent header. The corresponding signature base is: This results in the following Signature-Input and Signature header fields being added to the message under the label sig1:

Signature-Agent included present on the request This example presents a minimal signature using the rsa-pss-sha512 algorithm over test-request. The request contains a Signature-Agent header. The corresponding signature base is: This results in the following Signature-Input and Signature header fields being added to the message under the label sig2:

EdDSA Using Curve edwards25519 The test vectors in this section use the Ed25519 key defined in . This section include non-normative test vectors that may be used as test cases to validate implementation correctness.

Signature-Agent absent from the request This example presents a minimal signature using the ed25519 algorithm over test-request. The request does not contain a Signature-Agent header. The corresponding signature base is: This results in the following Signature-Input and Signature header fields being added to the message under the label sig1:

Signature-Agent included present on the request This example presents a minimal signature using the ed25519 algorithm over test-request. The request contains a Signature-Agent header. The corresponding signature base is: This results in the following Signature-Input and Signature header fields being added to the message under the label sig2:

Implementations This draft has a couple of public implementations. A demonstration server has been deployed to https://http-message-signatures-example.research.cloudflare.com/. It uses ed25519 example signing and verifying keys defined in .

Clients

• Chrome MV3 (TypeScript)
• Cloudflare Workers (TypeScript)
• Puppeteer script (JavaScript)
• Guzzle middleware (PHP)
• Python script (Python)
• Bot-Authentication (Python)
• HTPie plugin (Python)
• Web scrapers (scrapy/crawl4ai) (Python)
• HUMAN Verified AI Agents (Python)
• Linzer (Ruby)
• Rust binaries (Rust)

Servers

• Caddy plugin (Go)
• Cloudflare Workers (TypeScript)
• Apache module (C)

Test vectors

• In JSON format

Acknowledgments The editor would also like to thank the following individuals (listed in alphabetical order) for feedback, insight, and implementation of this document - Marwan Fayed, Maxime Guerreiro, Scott Hendrickson, Jonathan Hoyland, Alex Sandor Major, Mark Nottingham, Eugenio Panero, Lucas Pardue, Malte Ubl, Loganaden Velvindron, Tanya Verma.

Changelog v04

• Change Signature-Agent to a sf-dictionary
• Add security consideration around intermediary re-keying Signature-Agent
• Add target-uri as a valid replacement for @authority
• Add more contributors
• Add new implementations of architecture drafts
• Remove "purpose" field from web-bot-auth architecture example

v03

• Update linzer example url
• Fix section and name of status code 429
• Fix typos

v02

• Add response status code
• Add a few reference to improve readability
• Add consideration to sign additional headers
• Add use of TLS in security considerations
• Add RSASSA-PSS example
• Update acknowledgement section
• Reference new implementations: PHP, Python, Ruby, Rust
• Fix signature-agent in the arch diagram to use structured fields
• Fix test vectors to use signature-agent with structured fields
• Fix some typos

v01

• Add mandatory signing of Signature-Agent by clients if present
• Add test vectors for request with and without Signature-Agent
• Update example diagram to be correct
• Add security consideration about reverse proxy
• Update why origin may request a new signature
• Update nonce validation wording and global uniqueness
• Acknowledgements

v00

• Initial draft
• How to leverage HTTP Message Signature to sign request
• How to verify these Signature
• Define web-bot-auth tag to scope this signature
• Derive keyid using JWK Thumbprint
• High level Security and Privacy considerations