Definition and Problem Statement of consistent inter-domain routing and forwarding

The consistency of inter-domain routing and forwarding This document defines forwarding consistency as the forwarding of traffic exactly along the path planned by routing. Inconsistency forwarding means that the actual forwarding path is different from the path selected by other routers through routing. Routers route traffic depending on the routing information provided by routing protocols. Many factors, such as policy-based routing or traffic engineering, may result in that the forwarding path may be altered on the data plane by any routers and not conform to the routing path. In other words, the actual forwarding path is not verified by any consistent policies or routing algorithms. Therefore inconsistency forwarding may not match the intent of the network user and the route, leading to low quality, security risks or even unreachability. inconsistency forwarding is especially common in inter-domain scenarios. The AS_PATH attribute of BGP UPDATE records the AS number that it has passed through. Ideally, the traffic to the destination address should be forwarded along the AS number recorded in the AS_PATH. For purposes of load balancing, traffic analysis, and so on, the actual forwarding AS path can be affected by multiple entities and usually different from the propagation path of BGP UPDATE. In addition, the AS_PATH attribute is easy to be modified and may not reveal the actual propagation path of BGP UPDATE, which also contributes to inconsistency inter-domain forwarding. Inconsistency inter-domain forwarding causes the forwarding path of traffic in the Internet to be a black box for any AS. Worsely, the inter-domain forwarding path is not validated by any consistent policies or routing algorithms, which may incurs many forwarding risks such as traffic black hole, traffic loop, traffic detour and crossing the malicious AS, etc. Operators can only engineer inter-domain traffic based on simple consensus and best practices, burdening network maintenance and making it difficult to prevent these forwarding risks.

The complete forwarding AS path shows a simple case for inconsistency inter-domain forwarding. The comlpete forwarding AS path from AS1 to AS2 consists fo BGP paths and non-BGP paths. The non-BGP path from ASx to ASy, decided by non-BGP protocols, is invisible to AS1 and AS2. Similarly, the BGP path from ASy to AS2 is invisible to ASx and the BGP path from AS1 to ASx is invisible to ASy. If the two BGP paths contains the same AS, a loop occurs between AS1 and AS2. This loop cannot be detected or broken by any AS on the path. Forwarding consistency aims to empower BGP to open the forwarding black box in the Internet, enabling the actual forwarding path to be visible to the origin AS. It is unlikely to force all traffic to be forwarded according to the routing decision. However, it is essential to extend BGP to reveal the actual forwarding AS path or the non-BGP fators that affect the forwarding path. In fact, the community has already proposed lots of work along these lines, such as BPG Flowspec , Add-Path . The document aims to provide a risk analysis of inconsistency forwarding, summarize the reasons for inconsistency, introduce definition and potential scenarios of forwarding consistency.

The risk of inconsistency inter-domain forwarding The inconsistency forwarding result in the origin AS, possibly all ASes, not seeing the complete actual AS forwarding path to the destination AS. The reachability of BGP relay on simple consensus, e.g., valley-free principle, and best practices. After more than 50 years of practice, this principle is seems pretty solid. The complete AS path, which is not verified by routing principles and filters of the control plane of any AS, still has many problems. Specifically, the inconsistency inter-domain forwarding has the following risks:

Traffic blackhole: there is no corresponding route for the next-hop AS, resulting in a traffic black hole. For example, an AS that violates the valley-free principle redirects traffic that should be forwarded to the provider AS to another unrelated customer AS. Both BGP Flowspec and PBR support redirection of traffic on the data plane, which is invisible to the control plane.
Loop: the complete AS path that has not been checked by secure routing principles of any AS may lead to loops, e.g., the AS path before redirection and the AS path after redirection may contain the same AS.
Detour: the complete forwarding AS path is composed of multiple AS paths from different protocol, which may lead to unnecessary lengthening of AS path.
Malicious AS: the actual forwarding path is not visible to the origin AS, which may cause its traffic to pass through some insecure ASes it does not expect.
Non-optimal route: the AS may select the optimal route based on some preferred ASes. However, the actual forwarding AS path may not contain the AS it expectes.

Reasons for inconsistency inter-domain forwarding This document focuses on inconsistency forwarding resulting from control-plane and data-plane disagreement. The inter-domain forwarding that does conform to routing may be caused by traffic redirection, traffic engineering protocols, and ambiguous routing specifications, etc.

Traffic Redirection Due to load-balancing, anti-DDoS, etc., policy-based routing (PBR) or BGP Flowspec may be configured to redirect traffic on the data plane to a new next-hop AS which is different with the next-hop AS determined by AS_PATH in BGP UPDATE. In addition, ECMP optionally ignores comparing AS_PATH of different routes, resulting in load-sharing of traffic across multiple different next-hop ASes. However, BGP usually only advertises the optimal route outwardly. The forwarding AS path of the traffic after the above redirection is determined by the next-hop AS that not in the AS_PATH attribute, which is unknown to the origin AS and the upstream AS. The complete forwarding AS path consists of two parts: the AS path before redirection and the AS path after redirection. This path is not verified by any AS and may violate the valley-free principle or even contain duplicate ASes, causing traffic blackholes or loops.

Traffic engineering protocols Due to load-balancing, network optimization, etc., operators may utilize other protocols such as MPLS，SR to locally steer traffic to the specified AS path which is different from the AS_PATH in BGP UPDATE. The complete forwarding AS path is determined by BGP and TE protocols. It may include multiple segments, for example, the first segment is an AS path determined by BGP that starts with the origin AS, the second segment is a TE path and the last segment is still the AS path determined by BGP that ends with the destination AS. The complete forwarding path is actually transparent to the origin AS and is not verified by its filters, which may incur similar risks as redirection.

Ambiguous routing specifications Ambiguous routing specifications, such as route aggregation, convergence and redistribution, may also contribute to the inconsistency between inter-domain routing and forwarding. To minimize routing tables, route aggregation is widely used in IP networks. BGP route aggregation causes the ordered AS_SEQUENCE to be converted to the unordered AS_SET. AS_SEQUENCE and AS_SET which are different types of AS_PATH. AS_SET does not indicate the forwarding AS path of traffic, which can lead to the inconsistency between inter-domain routing and forwarding. Redistributing BGP routes into IGP can result in the loss of AS_PATH. Before advertised to the other AS, the route should be redistributed from IGP into BGP. The other AS that receives the route by BGP considers the destination AS of the route to be the redistribution AS. In fact, the complete AS path that the origin AS wishes to obtain is actually the AS path from it to the destination AS. Fortunately, this reditribution is too dangerous to be practiced.

Potential Use case The purpose of comformance forwarding is to enhance the ability of routing protocols to keep data-plane and control-plane alignment by ensuring that all forwarding behavior is controlled by the routing protocol or reflected in routing announcements. The primary goal of the routing protocol is to provide connectivity. With the emergence of hyperscale networks and diverse applications, routing protocols are given the ability to advertise more routing or forwarding information to reduce the burden on network management and provide better network quality. Existing routing protocols, expecially BGP, already support the advertisemnet of almost all routing and forwarding information. There is a lack of frameworks and requirements to gudie BGP to integrate all the information and open the black box of forwarding on the control plane. There are some scenarios where inter-domain forwarding consistency may be required:

Network Visualization

Visualization has been an enduring topic since the birth of the Internet. The forwarding behavior defined by the data plane makes visualization and predictability of the forwarding path even more challenging. Forwarding consistency allows the actual forwarding path of packets to be predicted based on route announcements. Current developments in BGP, such as BGP flowspec and extensions for TE, have demonstrated that route announcements can carry forwarding information that directly affects the path of packets on the data plane.

Intent-based routing

Intent-based routing provides flexible, scalable, and reliable end-to-end connectivity for multiple services accross the network domains by carrying multiple supported intent in routing protocols. Intent reflects the demand of application for transmission quality. Intent routing presupposes forwarding consistency. If inter-domain forwarding does not follow the routing, the intent may be not satisfied.

Source address validation

Source address validation based on RIB or FIB suffers from unavoidable improper pass or improper filter problems. Accurate source address validation verification relies on the actual forwarding path of the packet. However, inconsistency forwarding makes the high overhead of the discovery of the path unacceptable. Forwarding consistency can reduce the overhead, which contributes to TE and route security.

Requirements for inter-domain forwarding consistency All use cases require the inter-domain routing protocol, i.e., BGP, to learn deviating paths that are determined by non-BGP factors. Therefore, to get the complete actual forwarding AS path by BGP, there are two requirements:

Obtaining deviation AS paths that results from local non-BGP factors.
Advertising the deviation AS path and the steered flow by BGP

Requirement 1: Obtaining deviation AS paths Rotuers need to understand how non-BGP factors affect forwarding AS paths to learn deviating paths, which is vital to network visualization.

Obtaining redirection deviation AS paths A router redirecting traffic to a new next-hop AS generates a deviation AS path. In order to get the direction deviation AS path, the router first acquires the next-hop AS and the destination prefix of the redirection rule. The router then looks up the AS_PATH in Adj_RIBs_In from the next-hop AS by the destination prefix. The AS_PATH is the forwarding AS path after redirection, i.e., the redirection deviation AS path.

Obtaining deviation AS path from other protocols Some TE protocols that may direct the specific flow into pre-planned AS paths. Their destination prefix of the steered traffic is obtained in a similar way as in . In the egress router of the TE path, the Adj-RIBs-In is then looked up by the destination prefix to obtain the subsequent AS path. The complete forwarding path, therefore, is stitched together from the TE Path and other AS Paths controlled by BGP.

Requirement 2: Advertising the deviation path Advertising the devation path benefits other ASes not only in terms of network visualization but also in terms of optimal routing. The AS that generates the deviation AS path is obliged to advertise it to other ASes. For example, the AS that is configured with the redirection rule advertises the redirection deviation path to the origin AS. The origin AS then combines the AS path to the redirection AS and the redirection deviation AS path to get the complete forwarding AS path which is verified to be optimal or secure using the local AS_PATH filter. The path that passes through malicious ASes will not be selected by the origin AS. However, only the flow that matches the redirection rule will go through the deviation path, and the other missed flow is still forwarded along the original path planned by BGP. Therefor, the redirection AS should advertise the deviation path and the flow specification to other ASes.

Advertising the deviation path shows an example of advertising the devation path. As shown in the figure, AS4 advertises a BGP UPDATE to ASx along the AS path ([AS1,AS2,AS4]). AS2 is a redirection AS which redirect the packet routing to AS4 to AS3. As a result, the actual forwarding path of packets from AS1 to AS2 is [AS1, AS2, AS3, AS4]. So AS2 should add D_PATH to BGP UPDATE. D_PATH refers to the actual forwarding AS path through which the packet sent to AS4. The router generating the deviation AS path needs to advertise the non-BGP factors or the deviation AS path to other routers by the inter-domain routing protocol, i.e., BGP. In other words, inter-domain forwarding consistency requires that BGP announcements contain all possible forwarding behaviors. The routing algorithm integrates all possible forwarding behaviors and paths to select the optimal forwarding path. In short, routing announcements are subject to forwarding, but routing dictates the ultimate forwarding path.

Benefits Since the birth of the Internet, inter-AS TE has been a very complex and difficult task. One of the major reasons is that inter-domain routing and forwarding are inconsistency The AS_PATH attrtibute in BGP UPDATE only represents the propagation path of the route, which may not correspond to the actual forwarding path. The community has designed many complex protocols to plan the forwarding path and guarantee SLA, while routing protocols are usually utilized to get the basic reachability. If inter-domain forwarding conforms to inter-domain routing announcements, TE and routing security will be significantly simplified. Moreover, the ability of the origin AS to plan the forwarding AS path of its own path opens the black box of the Internet. Problems that have plagued the Internet for decades, such as visualization and troubleshooting, may be solved.

Management Considerations TBD

IANA Considerations TBD.

Security Considerations This document does not introduce any new security considerations.

Acknowledgements TBD