]> Huawei Technologies

China c.l@huawei.com

China Mobile

China liupengyjy@chinamoblie.com

Routing IDR CATS, BGP CATS (Computing-Aware Traffic Steering) is a traffic engineering approach that takes into account the dynamic nature of computing resources and network state to optimize service-specific traffic forwarding towards a service contact instance. This document defines the control plane BGP extension for CATS (Computing-Aware Traffic Steering).

Introduction When deploying edge clouds, multiple service instances might be deployed in multiple edge sites to provide equivalent function to end users. However, the resource of each edge site is limited due to only limited end users accessing to it locally. When burst traffic is generated by emergent events, the traffic should be balanced to multiple edge sites to provide better services to the end users. In order to provide better traffic steering among edge sites, a framework called CATS (Computing-Aware Traffic Steering) is proposed. CATS (Computing-Aware Traffic Steering) is a traffic engineering approach that takes into account the dynamic nature of computing resources and network state to optimize service-specific traffic forwarding. Various relevant metrics are defined in , which can be used in CATS. A general BGP extension of conveying computing metrics are defined in . It proposed a new Path Attribute, called Metadata Path Attribute, and some related sub-TLVs to carry the computing related metadata. This document defines the BGP extension for CATS control plane. Depending the deployment of CATS, there are two modes for CATS control plane, 1) Basic mode and 2) Sharing mode. For the basic mode, CATS can reuse the BGP extension defined in . For the sharing Mode, this document defines a new mechanism based on the basic extension defined in .

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document uses terms defined in . Some terms are listed below for clarification.

• Computing-Aware Traffic Steering (CATS): An architecture that takes into account the dynamic nature of computing resources and network state to steer service traffic to a service instance. This dynamicity is expressed by means of relevant metrics.
• Service: An offering that is made available by a provider by orchestrating a set of resources (networking, compute, storage, etc.).
• Service instance: An instance of running resources according to a given service logic.

Control Plane Modes Generally, in edge cloud, each service exclusively occupies some computing resources in the edge cloud. Therefore, in the CATS framework, when a service needs to synchronize the computing status of the occupied computing resource to the network, the computing status is transferred in a per-service manner. In other words, each service has its own computing status information release and control plane processing. This mainstream deployment mode is called the basic mode. In this mode, each service exclusively occupies computing resources, causing the price of cloud resources and the CATS scheduling may be high, which is more suitable for large-scale content/service providers such as OTT. However, the price of this deployment mode may be too high for small or medium-sized content/service providers, which prevents a large number of small- and medium-sized content/service providers from choosing CATS services. In order to allow more small- and medium-sized content/service providers to choose edge computing and CATS scheduling services, this document proposes a new mode, called sharing mode. In sharing Mode, multiple services can use the same shared resource and share the CATS scheduling service in the BGP control plane, reducing the price of purchasing edge cloud services and CATS scheduling services. This deployment mode also enables carriers to obtain more service providers. The detailed BGP extension of basic and sharing modes will be described in the following sections.

Basic Mode In the basic mode, each service is deployed independently, including occupying independent computing resources in edge site. The computing resources update is triggered by the service itself and independent with each other, for example, by using an independent BGP update message. In this mode, the computing metrics update and the CATS steering decision is implemented in a service-oriented way. In order to support 5G services in edge cloud, a general mechanism with BGP extensions is defined in . CATS framework basic mode can reuse the extensions defined in . However, in real implementation, only the basic extensions instead of all the extensions defined are required to be implemented. This document will specify the mandatory part and optional part of extensions in CATS implementation to support easer implementation and inter-operability test.

Mandatory Extensions This section specifies the mandatory extensions for CATS in implementation. In CATS, the fundamental features are reporting the computing metrics from the edge site to the network devices, such as the egress router of CATS overlay path, which is connected to the edge site. Therefore the basic and mandatory features are reporting the service related capability and available resource to the network devices. When the egress CATS forwarder receive the computing metrics, it can reuse the Service-Oriented Capability and Service-Oriented Available Resource sub-TLVs to distribute the metrics to the ingress CATS forwarder.

Service-Oriented Capability Capability information describes the total resources that can be used by a service, which is important in selecting the best service contact instance, so the distribution of this information is mandatory. This document reuses the Service-Oriented Capability sub-TLV in Metadata Path Attribute to distribute the capability information from an egress CATS forwarder to an ingress CATS forwarder. The format of Service-Oriented Capability sub-TLV is shown below for reference.

Service-Oriented Capability sub-TLV

The received capability information of a service is encoded by the egress CATS forwarder into the SO-CapValue field, and the sub-TLV will be encoded into the Metadata Path Attribute. The whole mechanism of encoding and processing the capability reuses the mechanism defined in .

Service-Oriented Available Resource Service-Oriented Available Resource sub-TLV is defined for distributing the real-time available resource of a service. The real-time available resource is vital for some low-latency service/applications which need the dynamic resource status to achieve on-time load balancing. Therefore, the distribution of available resource for a service is mandatory in CATS implementation. This document reuses the Service-Oriented Available Resource sub-TLV in Metadata Path Attribute to distribute the available resource information from an egress CATS forwarder to an ingress CATS forwarder. The format of Service-Oriented Available Resource sub-TLV is shown below for reference.

Service-Oriented Available Resource sub-TLV

The received available resource information of a service is encoded by the egress CATS forwarder into the SO-AvailRes field, and the sub-TLV will be encoded into the Metadata Path Attribute. The whole mechanism of encoding and processing the capability reuses the mechanism defined in . The above two sub-TLVs and the Metadata Path Attribute are required in CATS implementation.

Optional Extensions In order to support some enhanced features, the following optional extension can be implemented when implementing CATS.

Site Preference Site preference indicates the preference of each site when comparing service contact instances from different site. The preference value may be generated from several factors, such as the price of energy, renting fee of DC rack, etc. However, this is a site-level selection instead of service-level. Except the same site-level conditions, such as energy price, other service specific factors may be more important when selecting a better service contact instance from sites. In addition, the factors such as energy price can be added as a factor in generating the normalized metric value of Service-Oriented Capability and Available Resource. Therefore, Site Preference is an optional metric in CATS implementation, and the extension and processing refers to the Site Preference Index Sub-TLV defined in .

Site Physical Availability Index Similar to Site Preference, Site Physical Availability Index is a site-level metric, which is useful in batch processing of BGP updates. This is an enhancement of the CATS implementation, which can be implemented optionally.

Delay Prediction The end-2-end delay information is vital for time-sensitive service, such as 5G uRLLC services. However, it is not easy to detect the real-time delay in deployment. As per , Service Delay Prediction Index sub-TLV is defined for encoding the delay information. It includes two types of delay formats, one is for delay prediction index, another one is the NTP format of raw delay information. CATS implementation can reuse this mechanism. Distributing real-time delay information may bring too heavy burden to the BGP control plane, therefore this document defines it as an optional feature in CATS implementation. Furthermore, the delay information can be predicted by some raw metrics, such as the capability of GPU/CPU, which can be considered as factors when generating the normalized Service-Oriented Capability and Service-Oriented Available Resource metrics.

Raw Metadata Raw metadata is too complicated in standardization and implementation, therefore, this document does not recommend to implement the mechanism of Raw Metadata defined in . An implementation might choose to implement it. In the basic mode, all the extensions and mechanisms can reuse the extensions and mechanisms defined in , therefore, this document will not explain the details of processing any sub-TLVs.

Sharing Mode In the Basic mode, each service will occupy its own computing resources in the edge site. But the price of edge clouds might be too high to small- and medium-sized service providers, which may block them to deploy server closer to end users. Sharing Mode is a mode that allows different service providers to share the same computing resources when deploying services in edge sites. Since the computing resources are shared among different services, the price of the computing resources for each service can be reduced. When multiple services share the same computing resources, all the services will be influenced by the status of the shared computing resources. For example, in the Basic mode, when the available computing resources is running out, a C-SMA (CATS Service Metric Agent) can collect the metrics per-service and distribute them to the egress CATS forwarder. The egress CATS forwarder will redistribute the metrics in a per-service manner to ingress CATS forwarders. If there are 10 services are sharing the shared computing resources, then the C-SMA should collect 10 times for these 10 services respectively only because of one single metrics update of the shared computing resource. Furthermore, the egress CATS forwarder may generate 10 BGP update messages to distribute the update of the metrics to the ingress CATS forwarder though all the metrics of in the Metadata Path Attribute are the same. When different service's routes are using the same Path attributes, including the same Metadata Path Attribute, the 10 BGP update messages can be packed in to a single update message. However, when different service routes use different Path Attributes, a single update of the shared computing resources can trigger more than 1 update messages. Sharing mode provides a new mechanism to address this problem by turning service-oriented metrics update into resource-oriented metrics update or service-group-oriented metric update.

Workflow The Sharing mode mainly includes two phases of updating the computing resource metrics.

• Association establishment
• Resource-oriented/Service-group-oriented metrics update

In the first phases, a resource ID that identifies the shared computing resource, or a service group ID that identifies a group of service using a specific shared computing resource, needs to be added into the Service-Oriented Capability sub-TLV and Service-Oriented Available Resource sub-TLV. When the ingress CATS forwarder receives a BGP update message, it can learn the service ID (which is the prefix in the NLRI), and the resource ID or service group ID, and then build an association relation between them. In the second phase, the egress CATS forwarder may choose a possible method described below to distribute the metrics of the shared computing resources directly instead of sending multiple service oriented independent BGP update. When the ingress CATS forwarder receives the resource metric update, it will trigger all the processing of the associated services. In this way, only one BGP update message is needed for a single update for a shared resource, which can reduce N-1 (N is the number of the services that share the same shared computing resources) BGP update messages in the best case.

Extensions This section describes the extensions of Sharing mode. As mentioned above, in the association establishment phase, a resource ID that identifies the shared computing resource, or a service group ID that identifies a group of service using a specific shared computing resource, needs to be added into the Service-Oriented Capability sub-TLV and Service-Oriented Available Resource sub-TLV. The updated format of Service-Oriented Capability sub-TLV is shown below.

Service-Oriented Capability sub-TLV

Where,

• ResourceID (4 bytes): an ID that identifies the shared computing resource consumed by this service. 0 is reserved, meaning no specific shared resource is identified, and it MUST be ignored in processing.
• or ServiceGroupID (4 bytes): an ID that identifies a group of service that the service belongs to. All the service within this group use a specific shared computing resource. 0 is reserved, meaning this service is not sharing any resource with other services, therefore no service group is associated with it, so it MUST be ignored in processing.

Other fields are not changed in the sub-TLV. Similarly, the updated format of Service-Oriented Available Resource sub-TLV is shown below.

Service-Oriented Available Resource sub-TLV

Where,

• ResourceID (4 bytes): an ID that identifies the shared computing resource consumed by this service. 0 is reserved, meaning no specific shared resource is identified, and it MUST be ignored in processing.
• or ServiceGroupID (4 bytes): an ID that identifies a group of service that the service belongs to. All the service within this group use a specific shared computing resource. 0 is reserved, meaning this service is not sharing any resource with other services, there for no service group is associated with it, so it MUST be ignored in processing.

Other fields are not changed in the sub-TLV. When receiving the above sub-TLVs on the ingress CATS forwarder, if the association relation is not existed, the ingress can learn the association between the service ID (the prefix value in the NLIR) and the shared resource, which is identified by the resource ID or the service group ID. And then, the ingress CATS forwarder will refresh the metrics of the shared computing resource, which will trigger all the associated services to process this metrics update event. After the association relation is established, the later metrics update can use the following potential mechanisms to achieve less BGP update messages, to release the processing burden of BGP speakers. Editor note: Authors will update the draft to choose the best solution according to the WG conclusion.

Option 1: Reusing an Existing Update When the first metrics update is finished for a service, the ingress CATS forwarder learned the association relation, and register the shared computing resource update event for this service. When multiple services are sharing the same resources, the ingress CATS forwarder will add these services into a group identified by the resource ID or service group ID. For the following updates, as long as the ingress CATS forwarder receives an update message that contain a resource ID or service group ID, then the ingress updates the status of that shared resource or the associated service group using the shared resource, and trigger CATS processing for all the associated services. In this way, there is no need to send per-service BGP update message to the ingress. Instead, choosing only one service update with a resource ID/service group ID can trigger all the updates for services sharing the shared resources. Editor's note: In order to make the logic clearer, a flag can be added into Service-Oriented Capability and Service-Oriented Available Resource sub-TLVs, to identify the action of association establishment and batch update. The implementation is similar as the I-flag in Site Physical Availability Index Sub-TLV . This can be added in the future revision after WG discussion.

Option 2: Using a Specific Prefix Another solution is to use a specific configured or well-known prefix value to indicate that this update is for a shared resource in the Sharing mode instead of for a specific service/prefix. This specific prefix value can be configured in the CATS domain by operators, or this document can apply for a specific purpose of prefix for this. Editor's note: However, this is complicated than the option one, so editor will recommend the option 1.

Option 3: Using a New NLRI The above options try to reuse the existing solution, and NLRI, which is easier in implementation. However, in the Sharing mode, the computing resource metrics update has become a resource-oriented or service-group-oriented solution, therefore, a resource-oriented or service-group-oriented NLRI might help the implementation from separating the computing metrics and network metrics. A potential NLRI format is shown below.

NLRI for Resouce/Service Group

Where,

• length (1 byte): one byte length, indicates the length of the NLRI.
• SiteID (2 bytes): a 2 bytes SiteID identifying the ID of a site. Typically, a site can be an edge cloud site. the value comes from the Site-ID field in Site Physical Availability Index Sub-TLV .
• ResourceID (4 bytes): an ID that identifies the shared computing resource consumed by this service. 0 is reserved, meaning no specific shared resource is identified, and it MUST be ignored in processing.
• or ServiceGroupID (4 bytes): an ID that identifies a group of service that the service belongs to. All the service within this group use a specific shared computing resource. 0 is reserved, meaning this service is not sharing any resource with other services, there for no service group is associated with it, so it MUST be ignored in processing.

The AFI is recommended to be 1 or 2 meaning that it can be applied for service using IPv4 and IPv6 prefix, while the SAFI is TBD. However, it is possible to allocate a new AFI for this new type of NLRI. The Path attributes of this NLRI reuse the Metadata Path Attribute . When distributing a metrics update of a computing resource, the related sub-TLVs are included in the Metadata Path Attribute accordingly, such as the Service-Oriented Capability sub-TLV. Since the ResourceID/ServiceGroupID has been in the NLRI, the ResourceID/ServiceGroupID SHOULD be ignored when encoding into the sub-TLVs, and MUST be ignored in receiving. Because the update frequency of the computing metrics might be significantly higher than the existing metrics, it is reasonable to consider to put all computing metrics into a new NLRI, so that the filtering and route damping can be implemented in NLRI level. Editor's Note: However, standardizing, implementing and deploying a new NLRI is a long-term work, which might be hard to be successful in the industry. Authors would like to hear more voices on these three options.

Deployment Considerations TBD

Manageability Considerations TBD

Security Considerations This section will be updated in the future revision.

IANA Considerations This section will be updated according to the progress of the sharing mode.

Normative References Huawei Technologies China Mobile Orange Telefonica Juniper Networks, Inc. This document describes a framework for Computing-Aware Traffic Steering (CATS). Particularly, the document identifies a set of CATS components, describes their interactions, and exemplifies the workflow of the control and data planes. China Mobile Huawei Technologies Huawei Technologies This document defines the computing metrics used in Computing-Aware Traffic Steering. Futurewei Microsoft Azure Huawei Technologies Verizon China Mobile This draft describes a new Metadata Path Attribute and some Sub-TLVs for egress routers to advertise the Metadata about the attached edge services (ES). The edge service Metadata can be used by the ingress routers in the 5G Local Data Network to make path selections not only based on the routing cost but also the running environment of the edge services. The goal is to improve latency and performance for 5G edge services. The extension enables an edge service at one specific location to be more preferred than the others with the same IP address (ANYCAST) to receive data flow from a specific source, like a specific User Equipment (UE). In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings.

Acknowledgements The authors would like to thank Zongpeng Du, Huijuan Yao, Kehan Yao, Guofeng Qian, Haibo Wang, Xia Chen, Jianwei Mao, Zhenbin Li, Xinyue Zhang, Weier Li, and Linda Dunbar for their comments and suggestions.

Contributors Huawei Technologies

shihang9@huawei.com