]> Cloudflare

ot-ietf@thibault.uk

Web and Internet Transport Web Bot Auth bot authentication web security glossary automated agents Automated traffic authentication presents unique security challenges, constraints, and opportunities that impact all Internet users. This document seeks to collect terminology and examples within the space, with a specific focus on AI related technologies. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the Web Bot Auth Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction Agents are increasingly used in business and user workflows, including AI assistants, search indexing, content aggregation, and automated testing. These agents need to reliably identify themselves to origins for several reasons:

1. Regulatory compliance requiring transparency of automated systems
2. Origin resource management and access control
3. Protection against impersonation and reputation management
4. Service level differentiation between human and automated traffic

Current identification methods such as IP allow-listing, User-Agent strings, or shared API keys have significant limitations in security, scalability, manageability, and fairness. This document presents these examples, as well as possible paths to address them.

Motivation There is an increase in agent traffic on the Internet. Many agents choose to identify their traffic today via lists of IP Addresses and/or unique User-Agents. This is often done to demonstrate trust and safety claims, support allow-listing/deny-listing the traffic in a granular manner, and enable sites to monitor and rate limit per agent operator. However, these mechanisms have drawbacks:

1. User-Agent, when used alone, can be spoofed meaning anyone may attempt to act as that agent. It is also overloaded - an agent may be using Chromium and wish to present itself as such to ensure rendering works, yet it still wants to differentiate its traffic to the site.
2. IP blocks alone can present a confusing story. IPs on cloud plaforms have layers of ownership - the platform owns the IP and registers it in their published IP blocks, only to be re-published by the agent with little to bind the publication to the actual service provider that may be renting infra. Purchasing dedicated IP blocks is expensive, time consuming, and requires significant specialist knowledge to set up. These IP blocks may have prior reputation history that needs to be carefully inspected and managed before purchase and use.
3. An agent may go to every website on the Internet and share a secret with them like a Bearer from . This is impractical to scale for any agent beyond select partnerships, and insecure, as key rotation is challenging and becomes less secure as the consumers scale.

Using well-established cryptography, we can instead define a simple and secure mechanism that empowers small and large agents to share their identity.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Agent

An autonomous entity that perceives the environment and can take actions on behalf of users.

Bot

A type of agent that operates automatically, often performing repetitive tasks. Bots may identify themselves or attempt to mimic human behavior.

Origin (Server)

The primary server hosting the web content or service that an agent intends to access.

Application Firewall

Controls incoming traffic to an origin based on a set of rules. This may include but is not limited to IP filtering, User-Agent matching, or cryptographic signature verification.

Reverse proxy

An intermediary server that forwards client requests to the origin server, often performing functions like load balancing, authentication, or caching.

Browser

A client application used to access web content. Browsers may also be orchestrated.

Human

A physical person, like you and me.

Rate limit

A control mechanism that restricts the access of an Agent to a resource provided by an Origin Server. An Origin can decide to rate limit all connections from an individual Client, from a specific Provider, or to a specific resource. This may be a fixed number of requests, a budget, a time, a location, or legal requirements.

Unlinkability

A property ensuring that multiple interactions or credentials from the same agent cannot be correlated by the verifier.

Account

Persistent identifier of an entity to an origin. This requires a registration.

Registration

The creation of an identity. It can involve one time payment, a subscription, an account with user name/password, an age, a legal jurisdiction, others.

Issuer

An entity that generates and provides credentials to agents after the Attester has verified certain attributes.

Attester

An entity that evaluates an agent's characteristics or behavior and provides evidence to an Issuer to support credential issuance.

Verifier

An entity that validates the authenticity and integrity of a credential presented by an agent.

Web bot authentication categories We divide web bot authentication in three categories.

Identifying providers Organizations operating bots may need to authenticate their agents to access certain web resources. Authentication mechanisms can help distinguish legitimate bots from malicious ones. Examples:

• Web crawlers wanting to authenticate against origins such as search engines,
• Security companies that want to perform scans to identify malicious URLs,
• AI augmented queries that are looking to identify themselves to a set of newspapers.

User Account Identification Bots acting on behalf of registered users may require authentication to access user-specific data or services. Examples:

• Authenticating and authorizing a known user against particular resources, such as newspapers they have a subscription for,
• Most authorization use cases for and .

Attribute-Based Access In scenarios where full identification is unnecessary or undesirable, agents may present credentials that attest to specific attributes without revealing their identity. Examples:

• Add a signal to limit visual CAPTCHA challenge such as ,
• Gating access to a resource for longstanding users such as ,
• Using a search engine with a fixed number of requests such as ,
• Selective disclosure of a credential attribute (location, age) such as .
• Redeeming previously issued credits as in .

Ecosystem overview | Reverse Proxy +------>| Origin | / +---------------+ +-----+----+ / ^ +----------+/ ║ | Client + Trust +----------+\ ║ \ v \ +----------+ +-----+----+ '--->| Attester +----------->| Issuer | +----------+ +----------+ ]]> The ecosystem involves multiple actors: a credential issuer that requires an certain criteria to be passed via an attester, the client which can be a bot or human-mediated agent whose IP is unknown, and the web origin placed behind a reverse proxy that may be fronting its infrastructure. The issuer provides cryptographic credentials to the client, which are then linked to requests and optionally verified by proxies before reaching the origin. This chain allows for authentication without necessarily revealing identifying details to each intermediate.

AI agent use example Humans and bots often interact with origins indirectly via clients such as browsers, agents, or CLI tools. These clients handle requests, potentially traversing reverse proxies that manage TLS termination, DDoS protection, and caching. The rise of advanced browser orchestration blurs the line between human-driven and automated requests, making identifying traffic as automated or not increasingly ambiguous. | | | +---------+ | | | | +---------+ | | | | | +-------------------+------>| | .---+->| Agent | +---------+ | | | | | | +--->| Browser +----+------>| | +-------+ + | +---------+ +---------+ | | | +--------+ | Human +--+ | | | Reverse Proxy +--->| Origin | +-------+ | '------------------------------' | | +--------+ | | | | +---------+ | | '-------------------->| Browser +----------->| | +---------+ | | | | +---------+ | | | Bot +----------->| | +---------+ +---------------+ ]]> The attester/issuer roles could be filled by the AI company, reverse proxy, origin, or a third party. Origins need mechanisms to identify organizations, rate-limit individuals, and authenticate users without relying solely on client IP or heuristics presented in .

Web crawler example Search engines, web archivers, and security analysis tools operate bots as part of their operations to scan and retrieve content from a variety of sources. These automated agents are commonly referred to as crawlers. While their traffic is automated, it is often seen as desirable by origins. Legitimate crawlers contribute to improved SEO, enhanced security posture, and broader content reach through search indexing and archiving. Reliable authentication allows origins to differentiate these beneficial crawlers from malicious actors. In the framework provided in , the interaction for a web crawler can be described as follows: | Reverse Proxy +------>| Origin | | +---------+ | +---------------+ +----------+ '--------------------------' ]]> Similar to the , the attester/issuer roles could be filled by the crawling company, a reverse proxy, the origin, or a third party. For instance, the crawling company itself is often the most authoritative entity to attest to the legitimacy of its crawlers, issuing cryptographic credentials that identify them. Origins require robust mechanisms to identify these organizations and differentiate their traffic without solely relying on the limitations of IP addresses or User-Agent heuristics presented in . This enables them to apply granular access controls, rate limits, or preferential treatment based on the verified identity of the crawler.

Deployment considerations The security model includes several actors: credential issuers, attesters, clients (bots or agents), reverse proxies, and origin servers. The primary goals are to prevent impersonation, allow for credential revocation, support delegation and rotation, and maintain trust boundaries.

Public vs private presentation If the Issuer is also the Origin or its reverse proxy, it is possible to use shared secrets for verification. In cases where the issuer and verifier are different entities, asymmetric cryptography becomes necessary, allowing the bot to prove its identity using a public key infrastructure. Such work is being carried out in the CFRG and Privacy Pass working group with the following drafts

Presentation	Cryptography	Privacy Pass
Private
Private
Public/Private
Public
Public

Single vs multi show Some credentials may be designed for one-time use only (for anti replay or privacy reasons), while others can support multiple presentations through the use of cryptographic derivation techniques. This distinction affects privacy, scalability, and implementation complexity.

Transport Authentication tokens may be exchanged at different protocol layers and through different transports. Each may have different deployment, performance, and security guarantees. For TLS, we have seen and respectively addressing and . For HTTP, we see or , and respectively addressing and . fits as well for . Other methods have been seen such as leveraging a dedicated format on top of a JavaScript API. This is the case for W3C or the more recent . Focusing on AI specifically, it's worth mentioning two proponent protocol definition efforts:

• which follows . This means it allows for Basic, Bearer, API Keys, and . OpenAPI mentions using the registry, but there does not seem to be a definition for recent schemes such as , , or .
• uses as a resource server.

Round trip Protocols should strive to minimise the number of round trips between a client and the issuer, and between clients and the origin.

Key management and discovery

Catalog Just as there are registries to resolve IP address metadata, there are going to be registries to identify the owner of public key material. These are mentioned by and . The primary goal of these catalogs is to associate metadata with a public key, and the discovery of the associated metadata. They SHOULD have some sort of tamper resistance, to prevent the provider of a catalog providing incorrect information. As an analogy, one can think of , or the more recent effort in .

Submission / out-of-band Submission is also going to happen out-of-band. This is both for a practical reason, it is simpler than setting up a catalog, and for privacy reasons, given you don't have to expose information through a catalog.

On-path Discovery may happen on-path, that is when a request arrives from a client to an origin. This could be considered a form of trust-on-first-use. While the level of trust is low, it could be viable for certain deployments for which knowing all agents a-priori is not viable. This could be due to their multiplicity, a frequent change in involved actors, or an origin that is willing to review new changes manually for instance. Such discovery could be via an HTTP header containing a domain name with a well-known, a URL, a certificate, etc.

Format There are a multitude of Key and directory formats. These include but are not limited to JWKS, CWKS, Privacy Pass, Agent Card, and HTTP Message Signatures.

Security Considerations This glossary provides terminology for web bot authentication. While this document does not define or recommend specific protocols, terminology choices have direct security implications:

Impersonation Resistance

Clearly defined roles are essential for preventing entities from falsely claiming identities.

Credential Replay and Theft

Definitions such as help describe key mechanisms that mitigate the misuse of credentials if stolen.

Key Management

is key to protocol security, and has to be considered.

In addition, protocols should consider decentralization and end-user impact .

Privacy Considerations Authentication mechanisms should minimize the collection and exposure of personal data. Techniques like selective disclosure and unlinkability help protect user privacy. Protocols should refer to . Multiple protocols are also likely to be used in coordination: to identify an orgnization, then to identify the User-Agent, and possibly rate limit. It is important to consider the privacy of these layers together as well.

IANA Considerations This document has no IANA actions.

References Normative References This document offers guidance for developing privacy considerations for inclusion in protocol specifications. It aims to make designers, implementers, and users of Internet protocols aware of privacy-related design choices. It suggests that whether any individual RFC warrants a specific privacy considerations section will depend on the document's content. This document explains why the IAB believes that, when there is a conflict between the interests of end users of the Internet and other parties, IETF decisions should favor end users. It also explores how the IETF can more effectively achieve this. This document discusses aspects of centralization that relate to Internet standards efforts. It argues that, while standards bodies have a limited ability to prevent many forms of centralization, they can still make contributions that assist in the decentralization of the Internet. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References n.d. n.d. Apple, Inc. Apple, Inc. This document specifies the Anonymous Rate-Limited Credential (ARC) protocol, a specialization of keyed-verification anonymous credentials with support for rate limiting. ARC credentials can be presented from client to server up to some fixed number of times, where each presentation is cryptographically bound to client secrets and application-specific public information, such that each presentation is unlinkable from the others as well as the original credential creation. ARC is useful in applications where a server needs to throttle or rate-limit access from anonymous clients. MATTR MATTR Portage CryptID This document describes the BBS Signature scheme, a secure, multi- message digital signature protocol, supporting proving knowledge of a signature while selectively disclosing any subset of the signed messages. Concretely, the scheme allows for signing multiple messages whilst producing a single, constant size, digital signature. Additionally, the possessor of a BBS signatures is able to create zero-knowledge, proofs of knowledge of a signature, while selectively disclosing subsets of the signed messages. Being zero-knowledge, the BBS proofs do not reveal any information about the undisclosed messages or the signature itself, while at the same time, guaranteeing the authenticity and integrity of the disclosed messages. This document specifies an RSA-based blind signature protocol. RSA blind signatures were first introduced by Chaum for untraceable payments. A signature that is output from this protocol can be verified as an RSA-PSS signature. This document is a product of the Crypto Forum Research Group (CFRG) in the IRTF. This document describes an experimental protocol for publicly logging the existence of Transport Layer Security (TLS) certificates as they are issued or observed, in a manner that allows anyone to audit certificate authority (CA) activity and notice the issuance of suspect certificates as well as to audit the certificate logs themselves. The intent is that eventually clients would refuse to honor certificates that do not appear in a log, effectively forcing CAs to add all issued certificates to the logs. Logs are network services that implement the protocol operations for submissions and queries that are defined in this document. Most HTTP authentication schemes are probeable in the sense that it is possible for an unauthenticated client to probe whether an origin serves resources that require authentication. It is possible for an origin to hide the fact that it requires authentication by not generating Unauthorized status codes; however, that only works with non-cryptographic authentication schemes: cryptographic signatures require a fresh nonce to be signed. Prior to this document, there was no existing way for the origin to share such a nonce without exposing the fact that it serves resources that require authentication. This document defines a new non-probeable cryptographic authentication scheme. This document describes a mechanism for sender-constraining OAuth 2.0 tokens via a proof-of-possession mechanism on the application level. This mechanism allows for the detection of replay attacks with access and refresh tokens. n.d. Cloudflare This document describes an architecture for identifying automated traffic using [HTTP-MESSAGE-SIGNATURES]. The goal is to allow automated HTTP clients to cryptographically sign outbound requests, allowing HTTP servers to verify their identity with confidence. This document defines the terminology and interaction patterns involved in the deployment of Key Transparency (KT) in a general secure group messaging infrastructure, and specifies the security properties that the protocol provides. It also gives more general, non-prescriptive guidance on how to securely apply KT to a number of common applications. n.d. n.d. n.d. The OAuth 2.0 authorization framework enables a third-party application to obtain limited access to an HTTP service, either on behalf of a resource owner by orchestrating an approval interaction between the resource owner and the HTTP service, or by allowing the third-party application to obtain access on its own behalf. This specification replaces and obsoletes the OAuth 1.0 protocol described in RFC 5849. [STANDARDS-TRACK] This specification describes how to use bearer tokens in HTTP requests to access OAuth 2.0 protected resources. Any party in possession of a bearer token (a "bearer") can use it to get access to the associated resources (without demonstrating possession of a cryptographic key). To prevent misuse, bearer tokens need to be protected from disclosure in storage and in transport. [STANDARDS-TRACK] n.d. An Oblivious Pseudorandom Function (OPRF) is a two-party protocol between a client and a server for computing the output of a Pseudorandom Function (PRF). The server provides the PRF private key, and the client provides the PRF input. At the end of the protocol, the client learns the PRF output without learning anything about the PRF private key, and the server learns neither the PRF input nor output. An OPRF can also satisfy a notion of 'verifiability', called a VOPRF. A VOPRF ensures clients can verify that the server used a specific private key during the execution of the protocol. A VOPRF can also be partially oblivious, called a POPRF. A POPRF allows clients and servers to provide public input to the PRF computation. This document specifies an OPRF, VOPRF, and POPRF instantiated within standard prime-order groups, including elliptic curves. This document is a product of the Crypto Forum Research Group (CFRG) in the IRTF. Google Google Cloudflare Google This document specifies a blind RSA signature protocol that supports public metadata. It is an extension to the RSABSSA protocol recently specified by the CFRG. Discussion Venues This note is to be removed before publishing as an RFC. Discussion of this document takes place on the Crypto Forum Research Group mailing list (cfrg@ietf.org), which is archived at https://mailarchive.ietf.org/arch/search/?email_list=cfrg. Source for this draft and an issue tracker can be found at https://github.com/chris-wood/draft-amjad-cfrg-partially-blind-rsa. Apple, Inc. Apple, Inc. This document specifies the issuance and redemption protocols for tokens based on the Anonymous Rate-Limited Credential (ARC) protocol. Akamai Technologies Existing token types in privacy pass conflate attribution with rate limiting. This document describes a token type where the issuer attests to a set of properties of the client, which the client can then selectively prove to the origin. Repeated showings of the same credential are unlinkable, unlike other token types in privacy pass. This document defines an HTTP authentication scheme for Privacy Pass, a privacy-preserving authentication mechanism used for authorization. The authentication scheme specified in this document can be used by Clients to redeem Privacy Pass tokens with an Origin. It can also be used by Origins to challenge Clients to present Privacy Pass tokens. Apple Google This document defines a mechanism for TLS servers to request, and TLS clients to provide, Privacy Pass tokens as part of the Encrypted Client Hello in the TLS handshake. This creates a way to add support for anonymous attestation and rate-limiting to servers that are enforcing denial-of-service protections as part of processing TLS handshakes. n.d. This document specifies two variants of the two-message issuance protocol for Privacy Pass tokens: one that produces tokens that are privately verifiable using the Issuer Private Key and one that produces tokens that are publicly verifiable using the Issuer Public Key. Instances of "issuance protocol" and "issuance protocols" in the text of this document are used interchangeably to refer to the two variants of the Privacy Pass issuance protocol. Google Cloudflare, Inc. This document specifies Privacy Pass issuance protocols that encode public information visible to the Client, Attester, Issuer, and Origin into each token. n.d. n.d. n.d. n.d. Cloudflare Normally in TLS there is no way for the client to signal to the server that it has been configured with a certificate suitable for mTLS. This document defines a TLS Flag [I-D.ietf-tls-tlsflags] that enables clients to provide this hint. n.d.

Acknowledgments TODO acknowledge.

Changelog v01

• Improve wording from on-path submission (thanks to Christopher Patton)
• Add CFRG and Privacy Pass drafts within public/private presentation (thanks to Christopher Patton)
• Add web crawler example along with AI agent
• Add anonymous credit tokens draft (thanks to Sam Schlesinger)

v00

• Initial draft
• Overall ecosystem architecture
• Terminology
• Rough deployment considerations
• Reference multiple agent protocols