]> Cloudflare

ot-ietf@thibault.uk

bot authentication web security glossary automated agents Automated traffic authentication presents unique security challenges, constraints, and opportunities that impact all Internet users. This document seeks to collect terminology and examples within the space, with a specific focus on AI related technologies. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Source for this draft and an issue tracker can be found at .

Introduction Agents are increasingly used in business and user workflows, including AI assistants, search indexing, content aggregation, and automated testing. These agents need to reliably identify themselves to origins for several reasons:

1. Regulatory compliance requiring transparency of automated systems
2. Origin resource management and access control
3. Protection against impersonation and reputation management
4. Service level differentiation between human and automated traffic

Current identification methods such as IP allow-listing, User-Agent strings, or shared API keys have significant limitations in security, scalability, manageability, and fairness. This document presents these examples, as well as possible paths to address them.

Motivation There is an increase in agent traffic on the Internet. Many agents choose to identify their traffic today via lists of IP Addresses and/or unique User-Agents. This is often done to demonstrate trust and safety claims, support allow-listing/deny-listing the traffic in a granular manner, and enable sites to monitor and rate limit per agent operator. However, these mechanisms have drawbacks:

1. User-Agent, when used alone, can be spoofed meaning anyone may attempt to act as that agent. It is also overloaded - an agent may be using Chromium and wish to present itself as such to ensure rendering works, yet it still wants to differentiate its traffic to the site.
2. IP blocks alone can present a confusing story. IPs on cloud plaforms have layers of ownership - the platform owns the IP and registers it in their published IP blocks, only to be re-published by the agent with little to bind the publication to the actual service provider that may be renting infra. Purchasing dedicated IP blocks is expensive, time consuming, and requires significant specialist knowledge to set up. These IP blocks may have prior reputation history that needs to be carefully inspected and managed before purchase and use.
3. An agent may go to every website on the Internet and share a secret with them like a Bearer from . This is impractical to scale for any agent beyond select partnerships, and insecure, as key rotation is challenging and becomes less secure as the consumers scale.

Using well-established cryptography, we can instead define a simple and secure mechanism that empowers small and large agents to share their identity.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Agent

An autonomous entity that perceives the environment and can take actions on behalf of users.

Bot

A type of agent that operates automatically, often performing repetitive tasks. Bots may identify themselves or attempt to mimic human behavior.

Origin (Server)

The primary server hosting the web content or service that an agent intends to access.

Application Firewall

Controls incoming traffic to an origin based on a set of rules. This may include but is not limited to IP filtering, User-Agent matching, or cryptographic signature verification.

Reverse proxy

An intermediary server that forwards client requests to the origin server, often performing functions like load balancing, authentication, or caching.

Browser

A client application used to access web content. Browsers may also be orchestrated.

Human

A physical person, like you and me.

Rate limit

A control mechanism that restricts the access of an Agent to a resource provided by an Origin Server. An Origin can decide to rate limit all connections from an individual Client, from a specific Provider, or to a specific resource. This may be a fixed number of requests, a budget, a time, a location, or legal requirements.

Unlinkability

A property ensuring that multiple interactions or credentials from the same agent cannot be correlated by the verifier.

Account

Persistent identifier of an entity to an origin. This requires a registration.

Registration

The creation of an identity. It can involve one time payment, a subscription, an account with user name/password, an age, a legal jurisdiction, others.

Issuer

An entity that generates and provides credentials to agents after the Attester has verified certain attributes.

Attester

An entity that evaluates an agent's characteristics or behavior and provides evidence to an Issuer to support credential issuance.

Verifier

An entity that validates the authenticity and integrity of a credential presented by an agent.

Web bot authentication categories We divide web bot authentication in three categories.

Identifying providers Organizations operating bots may need to authenticate their agents to access certain web resources. Authentication mechanisms can help distinguish legitimate bots from malicious ones. Examples:

• Web crawlers wanting to authenticate against origins such as search engines,
• Security companies that want to perform scans to identify malicious URLs,
• AI augmented queries that are looking to identify themselves to a set of newspapers.

User Account Identification Bots acting on behalf of registered users may require authentication to access user-specific data or services. Examples:

• Authenticating and authorizing a known user against particular resources, such as newspapers they have a subscription for,
• Most authorization use cases for and .

Attribute-Based Access In scenarios where full identification is unnecessary or undesirable, agents may present credentials that attest to specific attributes without revealing their identity. Examples:

• Add a signal to limit visual CAPTCHA challenge such as ,
• Gating access to a resource for longstanding users such as ,
• Using a search engine with a fixed number of requests such as ,
• Selective disclosure of a credential attribute (location, age) such as .

Ecosystem overview | Reverse Proxy |------>| Origin | / +---------------+ +-----+----+ / ^ +----------+/ ║ | Client + Trust +----------+\ ║ \ v \ +----------+ +-----+----+ '--->| Attester |----------->| Issuer | +----------+ +----------+ ]]> The ecosystem involves multiple actors: a credential issuer that requires an certain criteria to be passed via an attester, the client which can be a bot or human-mediated agent whose IP is unknown, and the web origin placed behind a reverse proxy that may be fronting its infrastructure. The issuer provides cryptographic credentials to the client, which are then linked to requests and optionally verified by proxies before reaching the origin. This chain allows for authentication without necessarily revealing identifying details to each intermediate.

AI agent use example Humans and bots often interact with origins indirectly via clients such as browsers, agents, or CLI tools. These clients handle requests, potentially traversing reverse proxies that manage TLS termination, DDoS protection, and caching. The rise of advanced browser orchestration blurs the line between human-driven and automated requests, making identifying traffic as automated or not increasingly ambiguous. | | | +---------+ | | | | +---------+ | | | | | +-------------------+------>| | .---+->| Agent | +---------+ | | | | | | +--->| Browser +----+------>| | +-------+ + | +---------+ +---------+ | | | +--------+ | Human +--+ | | | Reverse Proxy +--->| Origin | +-------+ | '------------------------------' | | +--------+ | | | | +---------+ | | '-------------------->| Browser +----------->| | +---------+ | | | | +---------+ | | | Bot +----------->| | +---------+ +---------------+ ]]> The attester/issuer roles could be filled by the AI company, reverse proxy, origin, or a third party. Origins need mechanisms to identify organizations, rate-limit individuals, and authenticate users without relying solely on client IP or heuristics presented in .

Security goals and threat model The security model includes several actors: credential issuers, attesters, clients (bots or agents), reverse proxies, and origin servers. The primary goals are to prevent impersonation, allow for credential revocation, support delegation and rotation, and maintain trust boundaries.

Public vs private presentation If the Issuer is also the Origin or its reverse proxy, it is possible to use shared secrets for verification. In cases where the issuer and verifier are different entities, asymmetric cryptography becomes necessary, allowing the bot to prove its identity using a public key infrastructure.

Single vs multi show Some credentials may be designed for one-time use only (for anti replay or privacy reasons), while others can support multiple presentations through the use of cryptographic derivation techniques. This distinction affects privacy, scalability, and implementation complexity.

Transport Authentication tokens may be exchanged at different protocol layers and through different transports. Each may have different deployment, performance, and security guarantees. For TLS, we have seen and respectively addressing and . For HTTP, we see or , and respectively addressing and . fits as well for . Other methods have been seen such as leveraging a dedicated format on top of a JavaScript API. This is the case for W3C or the more recent . Focusing on AI specifically, it's worth mentioning two proponent protocol definition efforts:

• which follows . This means it allows for Basic, Bearer, API Keys, and . OpenAPI mentions using the registry, but there does not seem to be a definition for recent schemes such as , , or .
• uses as a resource server.

Round trip Protocols should strive to minimise the number of round trips between a client and the issuer, and between clients and the origin.

Key management and discovery

Catalog Just as there are registries to resolve IP address metadata, there are going to be registries to identify the owner of public key material. These are mentioned by and . The primary goal of these catalogs is to associate metadata with a public key, and the discovery of the associated metadata. They SHOULD have some sort of tamper resistance, to prevent the provider of a catalog providing incorrect information. As an analogy, one can think of , or the more recent effort in .

Submission / out-of-band Submission is also going to happen out-of-band. This is both for a practical reason, it is simpler than setting up a catalog, and for privacy reasons, given you don't have to expose information through a catalog.

On-path Discovery may happen on-path, that is when a request arrives from a client to an origin. This could be considered a form of trust-on-first-use. While the level of trust is low, it could be viable for certain use cases. Such discovery could be via an HTTP header containing a domain name with a well-known, a URL, a certificate, etc.

Format There are a multitude of Key and directory formats. These include but are not limited to JWKS, CWKS, Privacy Pass, Agent Card, and HTTP Message Signatures.

Security Considerations This glossary provides terminology for web bot authentication. While this document does not define or recommend specific protocols, terminology choices have direct security implications:

Impersonation Resistance

Clearly defined roles are essential for preventing entities from falsely claiming identities.

Credential Replay and Theft

Definitions such as help describe key mechanisms that mitigate the misuse of credentials if stolen.

Key Management

is key to protocol security, and has to be considered.

In addition, protocols should consider decentralization and end-user impact .

Privacy Considerations Authentication mechanisms should minimize the collection and exposure of personal data. Techniques like selective disclosure and unlinkability help protect user privacy. Protocols should refer to . Multiple protocols are also likely to be used in coordination: to identify an orgnization, then to identify the User-Agent, and possibly rate limit. It is important to consider the privacy of these layers together as well.

IANA Considerations This document has no IANA actions.

References Normative References This document offers guidance for developing privacy considerations for inclusion in protocol specifications. It aims to make designers, implementers, and users of Internet protocols aware of privacy-related design choices. It suggests that whether any individual RFC warrants a specific privacy considerations section will depend on the document's content. This document explains why the IAB believes that, when there is a conflict between the interests of end users of the Internet and other parties, IETF decisions should favor end users. It also explores how the IETF can more effectively achieve this. This document discusses aspects of centralization that relate to Internet standards efforts. It argues that, while standards bodies have a limited ability to prevent many forms of centralization, they can still make contributions that assist in the decentralization of the Internet. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References n.d. n.d. This document describes an experimental protocol for publicly logging the existence of Transport Layer Security (TLS) certificates as they are issued or observed, in a manner that allows anyone to audit certificate authority (CA) activity and notice the issuance of suspect certificates as well as to audit the certificate logs themselves. The intent is that eventually clients would refuse to honor certificates that do not appear in a log, effectively forcing CAs to add all issued certificates to the logs. Logs are network services that implement the protocol operations for submissions and queries that are defined in this document. Most HTTP authentication schemes are probeable in the sense that it is possible for an unauthenticated client to probe whether an origin serves resources that require authentication. It is possible for an origin to hide the fact that it requires authentication by not generating Unauthorized status codes; however, that only works with non-cryptographic authentication schemes: cryptographic signatures require a fresh nonce to be signed. Prior to this document, there was no existing way for the origin to share such a nonce without exposing the fact that it serves resources that require authentication. This document defines a new non-probeable cryptographic authentication scheme. This document describes a mechanism for sender-constraining OAuth 2.0 tokens via a proof-of-possession mechanism on the application level. This mechanism allows for the detection of replay attacks with access and refresh tokens. n.d. Cloudflare This document describes an architecture for identifying automated traffic using [HTTP-MESSAGE-SIGNATURES]. The goal is to allow automated HTTP clients to cryptographically sign outbound requests, allowing HTTP servers to verify their identity with confidence. This document defines the terminology and interaction patterns involved in the deployment of Key Transparency (KT) in a general secure group messaging infrastructure, and specifies the security properties that the protocol provides. It also gives more general, non-prescriptive guidance on how to securely apply KT to a number of common applications. n.d. n.d. n.d. The OAuth 2.0 authorization framework enables a third-party application to obtain limited access to an HTTP service, either on behalf of a resource owner by orchestrating an approval interaction between the resource owner and the HTTP service, or by allowing the third-party application to obtain access on its own behalf. This specification replaces and obsoletes the OAuth 1.0 protocol described in RFC 5849. [STANDARDS-TRACK] This specification describes how to use bearer tokens in HTTP requests to access OAuth 2.0 protected resources. Any party in possession of a bearer token (a "bearer") can use it to get access to the associated resources (without demonstrating possession of a cryptographic key). To prevent misuse, bearer tokens need to be protected from disclosure in storage and in transport. [STANDARDS-TRACK] n.d. This document defines an HTTP authentication scheme for Privacy Pass, a privacy-preserving authentication mechanism used for authorization. The authentication scheme specified in this document can be used by Clients to redeem Privacy Pass tokens with an Origin. It can also be used by Origins to challenge Clients to present Privacy Pass tokens. Apple Google This document defines a mechanism for TLS servers to request, and TLS clients to provide, Privacy Pass tokens as part of the Encrypted Client Hello in the TLS handshake. This creates a way to add support for anonymous attestation and rate-limiting to servers that are enforcing denial-of-service protections as part of processing TLS handshakes. n.d. n.d. n.d. n.d. Cloudflare Normally in TLS there is no way for the client to signal to the server that it has been configured with a certificate suitable for mTLS. This document defines a TLS Flag [I-D.ietf-tls-tlsflags] that enables clients to provide this hint. n.d.

Acknowledgments TODO acknowledge.