Problem Statement and Requirements of end-to-end CATS

Introduction Based on the migration of computing resources towards the edge and the presentation of distributed and heterogeneous deployment patterns, the boundaries between computing resources and the underlay network are no longer exactly separated. Computing-related services and corresponding scheduling schemes have gained their popularity and necessity in current and future circumstances. Considering the large number of future service instances and the dynamic and continuous nature of computing state variations, reflecting detailed instance information in the system will result in an incalculable number of entries and computational complexity. This will be catastrophic, especially for a circumstance of distributed control plane. As illustrated in , the Ingress CATS-Router steers service-specific traffic along a CATS-computed path that leads to an Egress CATS-Router that connects to the most suitable edge site that host the service contact instance selected to satisfy the initial service demand. In some cases, the choice of the service contact instance may be left open to the Egress CATS-Router and the Egress CATS-Router selects a service contact instance using its knowledge of service and network capabilities as well as the current load as observed by the CATS router among other considerations. Therefore, the end-to-end CATS process is implemented in a hierarchical manner. A corresponding hierarchical two segment routing scheme is also proposed and analyzed in which separates the service routing path into two segments regarding the highly dynamic computing status. Aggregated per-site computing related metrics appears as a privileged option scalability-wise. High frequency of computing status updates is restrained in a higher level decision region which is the GCRS segment described in this draft. The introduction of hierarchical segment routing for end-to-end CATS brings about the following significant superiority:

The number of entries in the control plane is decreased while the computational complexity of path selection is also reduced.
Relatively dynamic computing status of any explicit instance is separated from a decision region in a higher level. The refresh of entries in the FIB is also suppressed.

Comparatively, in a non-hierarchical scheme, a specific instance may be directly selected in a sole decison. Thus, a qualitative comparison between hierarchical and non-hierarchical schemes is shown below.

Comparison between Hierarchical and Non-hierarchical Schemes However, a hierarchical segment routing implys a multi-device decision-making manner which leads to possible problems:

The explicit information of multiple service instances is blurred and becomes invisible. A cohesive set of information which represents a whole cloud sites or a cluster of service instances must be properly designed. A global decision-making requires correctness and consistency.
As a general problem in common distributed systems, global updates of status variations require a period of notification time. Thus, a microloop problem exists similar to IGP deployed conditions in conventional IP networks.

Thus, this draft proposes a problem statement and incremental requirements of hierarchical segment routing schemes in the end-to-end CATS process.

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Terminology

GCRS: Global Computing Resource and Service Status
FIB: Forwarding Information Base
IGP: Interior Gateway Protocol
SR: Segment Routing
SRv6: Segment Routing over IPv6

Problem 1: Design of a Cohesive Scheme to Make Consistent Decisions As shown below, Instance 1 to Instance 7 locate at PE 1 to PE 3 respectively and their explicit information is displayed with typical computing attributes of CPU cores, memory remains and load.

Primitive Metadata with a Hierarchical Scheme Without a hierarchical scheme, PE 1, PE 2 and PE 3 learn and maintain the information of each service instance as shown below. Entries generated and maintained at PE 1, PE 2 and PE 3 are completely identical. Suppose a computing-related service is sensitive to a memory attribute and other constraints are temporarily ignored, then Instance 2 should be selected as the most appropriate instance.

Explicit Entries without a Hierarchical Scheme Then, a hierarchical scheme is applied and suppose a summation method is utilized to express the aggregated information of all instances connected to a PE while explicit and detailed information is concealed. Entries maintained at PE 1, PE 2 and PE 3 vary from each other. Suppose a service request accesses at PE 3, the request is steered to PE 1 according to the entries stored at PE 3. Similar decision procedures are made at PE 1 and PE 2, then the request is steered between PE 1 and PE 2 continuously which generates a permanent loop.

Cohesive Entries with a Hierarchical Scheme <----------- ]]> Based on the above example, inappropriate aggregation algorithms lead to inconsistent decisions among multiple PEs in the global region. The fundamental reason is that the application of summation leads to the disappearance of comparability between aggregated information and detailed information and a summation value of a cluster of instances is always preferred and thus selected. Therefore, a requirement is proposed for Problem 1 that the cohesive information needs to be comparable to the information of any explicit single instance. In the condition illustrated in Figure 2, taking the maximum value may be an appropriate method to unify the decision-making process. No matter a service request accesses at PE 1, PE 2 or PE 3, a forementioned loop problem is avoided.

Adjusted Cohesive Entries with a Hierarchical Scheme Instance 2 PE 2-->PE 1-->Instance 2 PE 3-->PE 1-->Instance 2 ]]>

Problem 2: Lack of Detailed Information Affects Decisions Undoubtedly, there is an information gap to utilize aggregated information to generally represent the ability among multiple instances at a PE. Therefore, profound analysis is required to determine whether decisions made based on aggregated information can adapt to various scheduling strategies. Taking a preferred strategy and a balanced strategy as examples, the use case and analysis are presented below.

Average Value as the Cohesive Information | PE 3 | +---+--+---+ / \ / \ / \ / \ / \ +----------+ +----------+ | PE 1 +--+ PE 2 | +----------+ +----------+ At PE 1: At PE 2: Instance 1 100G Instance 4 160G Instance 2 100G Instance 5 200G Instance 3 400G Instance 6 300G Average 200G Average 220G ]]> Under a preferred strategy, suppose an average value is selected to represent the instances behind a PE, when a service request accesses at PE 3, it finds that the performance of PE 2 seems better than PE 1 since the aggregated value is higher at PE 2. However, the 'best' instance which is Instance 3 locates at PE 1. Therefore, a maximum value or a minimum value is suggested to be the aggregated information to adapt a preferred strategy. However, practical problems may be sophisticated. In a more complicated occasion below, more attributes are required to expose to fulfill a multi-factor optimization with constraints. As shown below, under scheme 1, scheme 2 and scheme 3, the traffic is misled to the incorrect PE with the knowledge of the exposed information, however, Instance 6 and Instance 7 prove to be the 'best' respectively which satisfies the requirement and maintains the most memory resource with a global vision. Taking scheme 1 as an example, a computing related service requires the end-to-end delay is no more than 50ms and it is reckoned as a memory sensitive service. The best value among instances for each attribute is selected as the aggregated value. Thus, 400G and 20ms represent the performance at PE 1 while 260G and 30ms represent the performance at PE 2. Referring to the aggregated value, PE 1 seems to be the 'better' selection. However, Instance 3 at PE 1 fails to satisfy the service requirement while Instance 1 and Instance 2 have lower memory values than Instance 6 at PE 2.

Cohesive Entries with Multiple Factors | PE 3 | +----------+ / \ / \ / 10ms \ 10ms / \ / \ +----------+ +----------+ | PE 1 +--------+ PE 2 | +----------+ +----------+ At PE 1: At PE 2: Instance 1 100G,20ms Instance 4 200G,30ms Instance 2 220G,20ms Instance 5 200G,30ms Instance 3 400G,40ms Instance 6 260G,30ms * Scheme 1 400G,20ms * 260G,30ms Best Memory, Best Delay Scheme 2 240G,27ms * 220G,30ms Average Memory, Average Delay +----------+ Access---->| PE 3 | +----------+ / \ / \ / 10ms \ 10ms / \ / \ +----------+ +----------+ | PE 1 +--------+ PE 2 | +----------+ +----------+ At PE 1: At PE 2: Instance 1 100G,20ms Instance 4 200G,30ms Instance 2 220G,20ms Instance 5 200G,30ms Instance 3 400G,40ms Instance 6 260G,30ms Instance 7 350G,30ms * Scheme 3 400G,40ms 260G,30ms * Instance with Best Memory ]]> Since the aggregated information shields explicit and detailed entries, it is difficult to locate the 'best' instance under a preferred strategy. However, appropriate aggregation algorithm can indicate part of the satisfied instances which could adapts to a balanced strategy. It is suggested that an aggregation algorithm should be designated with the knowledge of distribution of computing resources and corresponding scheduling strategies. With future considerations, the number of service instances may increase to a large amount, the explicit information of running status may be displayed in a continous manner. 50,000 instances with load from 40% to 60% for instance. With the instances sorted, the values in the 'best' section may be 40%, 40.1%, 40.2%, and a great number of similar values. Thus, it is difficult to distinguish and it also may not be neccessary to select the 'best' one. Under the mentioned circumstances, a hierarchical method with aggregation algorithms introduced can adapt well.

Problem 3: Microloop Caused by Interruptions As show below, suppose a service client accesses at PE 2 and Instance 2 located at a cloud site connected to PE 1 loses efficacy at a sudden. This event has not yet been notified to PE 2 in a timely manner which leads to a mistake that PE 2 still steers the traffic to PE 1 which is still reckoned as the best selection. However, whenever PE 1 discovers that Instance 2 becomes invalid and updates its FIB, PE 2 turns out to be the most appropriate choice decided at PE 1. Thus, the traffic is steered back to PE 2 which forms a microloop.

Interrupt Notification with Time Difference +----------+ | PE 1 +--------------------------+ PE 2 +<-------- Access +----++----+ <------------------- +----++----+ || || || || (---++---) (---++---) (--- ---) (--- ---) ------- -------- ------- -------- (+------------+ ) (+------------+ ) ( | Instance 1 | 4C,160G,40%. ) ( | Instance 4 | 6C,100G,20%. ) ( +------------+ ) ( +------------+ ) ( +------------+x ) ( +------------+ ) ( | Instance 2 |x 8C,300G,60%. ) ( | Instance 5 | 8C,200G,20%. ) ( +------------+x ) ( +------------+ ) ( +------------+ ) ( +------------+ ) ( | Instance 3 | 6C,200G,50%. ) ( | Instance 6 | 8C,250G,80%. ) (+------------+ ) (+------------+ ) ----------------------------- ----------------------------- ]]> Without a hierarchical scheme, the destination is explicitly indicated with a single decision process. Without introducing or configuring instance or path protection schemes, packets will be directly discarded. Compared with microloop occured in IGP, a microloop mentioned here has similarities. Distributed status storage and disorderly convergence lead to inconsistent decision-making among distributed devices. SR provides a scheme to eliminate potential loops in the network with minimal impact on the network, creating an acyclic SRv6 Segment List for instance. However, unlike conventional problems and solutions, an explicit destination is not determined within previous decisions in a hierarchical scheme. Therefore, explicit routing path indicated by SR is not enough to solve a hierarchical segment routing problem. Actually, a higher level decision has been made within the last entry lookup. Thus, entries should be distinguished according to their gradation in the overall hierarchical scheme. And finally a requirement is proposed for Problem 3 that hierarchical entries should be designed and a corresponding forwarding behaviour should be introduced to adapt the forementioned entries.

Incremental Requirements for end-to-end CATS As illustrated in , the requirements of CATS are listed. As analyzed in this draft, the incremental requirements for a hierarchical segment routing scheme for CATS are further concluded as follows:

The cohesive information calculated by an aggregation algorithm SHOULD be comparable to a explicit entry of any service instance to guarantee the consistency and basic correctness of decision-making process.
The aggregation algorithm SHOULD be appropriately designed according to the distribution of computing resources and a corresponding applied scheduling strategy to reflect the performance and compatibility of local instances. Qualified service instances MUST be distinguished.
Hierarchical forwarding information entries SHOULD be separated within the lookup process and a corresponding hierarchical forwarding mechanism SHOULD be introduced accordingly.

Security Considerations TBA.

Acknowledgements TBA.

IANA Considerations TBA.