Futurewei

Dallas, TX USA ldunbar@futurewei.com

Huawei

USA shares@ndzh.com

Oracle

California USA kmajumdar@microsoft.com

Arrcus

USA robert@raszuk.net

Arista

USA venkit@arista.com

The document describes the BGP mechanisms for SD-WAN edge node property discovery. These mechanisms include a new tunnel type and subTLVs for the BGP Tunnel-Encapsulation Attribute [RFC9012] and set of NLRI for SD-WAN underlay information. In the context of this document, BGP Route Reflector (RR) is the component of the SD-WAN Controller that receives the BGP UPDATE from SD-WAN edges and in turns propagates the information to the intended peers that are authorized to communicate via the SD-WAN overlay network. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in when, and only when, they appear in all capitals, as shown here.

BGP [RFC4271] can be used as a control plane for a Software Defined Wide Area Network (SD-WAN) to support Secure VPNs or simply to support a set of Secure links in a network. A brief definition of these two use cases is given in this section. Section 2 describes the BGP framework in terms of NLRIs supported, example topologies, and objectives for the BGP mechanisms. Section 3 describes the BGP mechanisms, and section 4 describes error handling for this mechanism. The BGP mechanisms for SD-WAN support requires a route reflector (RR) to have a secure connection to each BGP Peer participating in the BGP infrastructure support the SD-WAN. The establishiment of a secure connection between each BGP Peer and the RR is outside the scope of this specification. This document describes the BGP mechanisms for an SD-WAN edge nodes to established a BGP peering with other SD-WAN edge nodes, and pass information in order to establish and update secure overlay tunnels [Net2Cloud]. These mechanisms include a new tunnel type and subTLVs for the BGP Tunnel-Encapsulation Attribute [RFC9012] and a set NLRIs for SD-WAN underlay information.

A SD-WAN network defined in [MEF70.1] and [MEF70.2], refers to a policy-driven network over multiple heterogeneous underlay networks to get better WAN bandwidth management, visibility, and control. This document refers to this network as a SD-WAN Secure L3VPNs. the defines the five requirements for BGP usage in an SD-WAN Secure L3VPN networks and 3 scenarios for deployment. The five requirements defined are the following: Support for SD-WAN segmentation - that allows edge noded connected over tunnels to support VPNs. These VPNs can be MPLS VPNS (, with VRFs, MPLS L2VPN , , L3VPN , or IPsec tunnels. Support for Client Services interfacs on edge nodes (e.g. SD-WAN UNI [MEF70.1]), WD-WAN Traffic Segmentation, Zero Touch Provisioning, and Constrained Propagation of SD-WAN Edge Properities by BGP infrastructure using a Route Reflector. describes the three scenarios for SD-WAN networks comprised of a mixture of tunnels over private and public networks: Homogeneous Encripted SD-WAN - where edge nodes encrypt data prior to entering the SD-WAN, Differential Encrypted SD-WAN - where edge nodes encrypt traffic traversing public networks, but do not encrypt traffic over private secure networks, Private VPN PE based SD-WAN - where an existing private VPN is expanded by adding extra ports over the public internet. provides descriptions on the the use of SD-WAN technology for L3VPNS, the provisioning of SD-WAN nodes, and example BGP topologie and SD-WAN forwarding mechanisms. This document assumes the reader is familar with SD-WAN use case described in .

defines a BGP mechanisms that links routes (prefix and Next Hop) to a specific tunnels using a specific encapsulation. The SD-WAN Secure Links Topology uses a single Hybrid logical link on a SD-WAN Peer to represent multiple underlay topology links. The SD-WAN peer distributes IPsec security association (IPsec-SA) related information regarding the hybrid link or individual underlay links. The traffic is routed via normal IP v4/v6 forwarding without any VPN addition. The SD-WAN Secure Links provides some link security for some simple cases of the three scenarios from [SD-WAN-BGP-USAGE] that do not require L3VPN addresses (RD, prefix).

The following acronyms and terms are used in this document: Authorized BGP peer are Peers which a particular BGP Peer has been configured by local policy to connect to. Off-Premises Data Centers that usually host applications and workload owned by different organizations or tenants. Color Extended Community defined in [RFC9012]. Used interchangeably with SD-WAN controller to manage SD-WAN overlay path creation/deletion and monitor the path conditions between sites. Customer (Edge) Premises Equipment Virtual Private Secure network formed among CPEs. This is to differentiate such VPNs from most commonly used PE-based VPNs discussed in [RFC4364]. Encapsulation Extended Community defined in [RFC9012]. IPsec Security Association. Multi-Protocol Network Layer Reachability Information (MP_REACH_NLRI) Path Attribute defined in [RFC4760]. Route Reflector [RFC4456 Route Target Extended Community [RFC4360] IPsec Security Association An overlay connectivity service that optimizes transport of IP Packets over one or more Underlay Connectivity Services by recognizing applications (Application Flows) and determining forwarding behavior by applying Policies to them. [MEF-70.1][MEF-70.2] can be the SD-WAN edge node address, a WAN port address (logical or physical) of a SD-WAN edge node, or a client port address. A single logical tunnel that combines several links of different encapsulation iinto a single tunnel. This logical tunnel may exist as part of a SD-WAN Secure L3VPN or simply be a SD-WAN secure link for a flat network. The mesh of SD-WAN secure tunnels that support the uses cases defined by and [MEF-70.1][MEF70.2]. BGP transmits information about SD-WAN Hybrid tunnels in the SD-WAN Secure L3VPN network by sending L3VPN AFI/SAFI with a TEA with a tunnel type of SD WAN Hybrid Tunnel. Information about the IPsec Security Association (IPsec-SA) for the SD-WAN underlay for these SD WAN Hybrid tunnels is passed in the SD-WAN NLRI (AFI 1/74) with a TEA with SD-WAN Hybrid Tunnel. This document defines the BGP mechanisms for the SD-WAN Secure L3VPN, but the defines the requirements, scenarios, provisioning model, examples of normal BGP flows, and SD-WAN forwarding. A mesh of SD-WAN Hybrid tunnels may connect several SD-WAN edge nodes in a flat (non-VPN) network. The SD-WAN Secure Links use case does not support VPN identification, and applies only a simple creation of secure link with support for passing IPsec-SA information regarding the SD-WAN Hybrid Tunnel. BGP sends a Unicast v4/v6 NLRI (AFI/SAFI 1/1 and 2/1) with a TEA for a SD-WAN Hybrid tunnel to announce the existance of the link. Information about the IPsec Security Association (IPsec-SA) for the SD-WAN underlay for these SD WAN Hybrid tunnels is passed in the SD-WAN NLRIs (AFI/SAFI 1/74 and 2/74) with a TEA with SD-WAN Hybrid Tunnel Tunnel Encapsulation Path Attribute [RFC9012] Virtual Private Network VPN Routing and Forwarding instance Wide Area Network The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC8174] when, and only when, they appear in all capitals, as shown here.

This section provides a framework to describe the overall solution parts based on the a reference diagram shown in Figure 1. The framework covers the following: client routes supported, example Topologies, objectives of the SD-WAN Edge discovery, comparison of SD-WAN Secure VPNs with Pure IPsec VPNS, and what SD-WAN segmentation means in this SD-WAN context.

Tunnel Encapsulation may be attached to the prefixes from the Unicast v4 and v6 ((AFI/SAFI 1/1 and 2/1), and L3VPNs for v4 and v6 (AFI/SAFI 1/128 and 2/128) (per , ). The document defines SD-WAN Secure L3VPN as the SD-WAN Secure VPN defined in , [MEF-70.1], and [MEF70.2]. The SD-WAN Secure L3VPN requires the L3VPN for IPv4 and IPv6 be expanded to support SD-WAN Hybrid Tunnels and the passages of IPsec-SA information for the underlay tunnels that support SD-WAN Secure L3VPN. This document defines the SD-WAN Secure Links as links in network that uses SD-WAN tunnels to create secure Hybrid SD-WAN tunnels between SD-WAN End Nodes. The SD-WAN Secure Links application does not support identification of a VPN via L3VPN NLRI. The SD-WAN end node uses BGP to pass Unicast v4/v6 prefixes ((AFI/SAFI 1/1 and 2/1) routes with TEA with a SD-WAN Hybrid tunnel TLV. The SD-WAN secure links application also passes the IPSec-SA information for the underlay tunnels in BGP. This document specifies the BGP mechanism for SD-WAN Secure L3VPN and SD-WAN Secure Links.

This section describes how the topologies for SD-WAN Secure L3VPN and SD-WAN Secure Links can be served by the BGP SD-WAN logical Tunnel links.

This section summarizes the BGP requirements from the following three scenarios from can be handled by a BGP control plane using BGP Tunnel-Encapsulation attribute : Homogeneous Encripted SD-WAN, Differential Encrypted SD-WAN, and Private VPN PE based SD-WAN. Based on , it is easy to see how Figure 1 matches scenario 1 (Homogeneneous Encrypted SD-WAN) and scenario 2 (Differential Encrypted SD-WAN). Recasting Figure 1 as a logical BGP peering topology results in a BGP topology between PEs and CN (CE at customer site) results in Figure 2. Scenario 3 (Private VPNs based on SD-WAN) from aligns with the Figure 2. Hybrid SD-WAN tunnel infrastructure requires that the PEs (C-PE1, C-PE2, C-PE3, and C-PE4) have an existing topology that the Hybrid SD-WAN tunnel overlays. These PEs use local policy to sending the approprite traffic across the appropriate network based on local policy. Sample Topology

Logical BGP Topology for SD-WAN

Note that the Figure 1 SD-WAN node BGP peers (C-PE) are connected in the underlay by both trusted VPNs and untrusted public networks. For trusted VPNs, IPsec Security associations may not be set-up. In some circumstances, some SD-WAN node peers may be connected only by untrusted public networks. For the traffic over untrusted networks, IPsec Security Associations (IPsec SA) must be established and maintained. If an edge node has network ports behind a NAT, the NAT properties need to be discovered by the authorized SD-WAN peers.

Suppose the SD-WAN network topology from Figure 1 removes the L3VPN links. Instead the links are simply a combination of L3 WAN Networks (unsecured) and physically secure direct L2 and L3 links. The topology in Figure 1 becomes the underlay physical topology in Figure 3. The unsecured links need IPSec encryptions so the IPsec links must be configured. An SD-WAN Hybrid tunnel allows connections between the C-PEs (C-PE1, C-PE2, C-PE3, and C-PE) to be a single logical link. Therefore, the logical topology in Figure 3, reduces to the SD-WAN logical topology shown in Figure 2. Sample Topology

For both the SD-WAN Secure L3VPN network and the SD-WAN Secure Links, the BGP Peers are assumed to be connected to a Route Reflector via secure link. For an SD-WAN deployment with multiple RRs, it is assumed that there are secure connections among those RRs. How secure connections are established between the BGP Peer and the RR, or between the multiple RRs is out of the scope of this document. The existing BGP UPDATE propagation mechanisms control the edge properties propagation among the RRs. For some environments where the communication to RR is highly secured, [RFC9016] IKE-less can be deployed to simplify IPsec SA establishment among edge nodes.

The objectives of SD-WAN edge discovery for an SD-WAN edge node are to discover its authorized BGP peers and each peer's associated properties in order to establish secure overlay tunnels [Net2Cloud]. The attributes to be propagated for the SD-WAN Secure L3VPN case are: the SD-WAN (client) VPN information, the attached client routes per VPN, and the properties of the underlay networks over which the client routes can be carried. Like any VPN networks, the attached client routes belonging to specific SD-WAN VPNs can only be exchanged with the SD-WAN peer nodes authorized to communicate. The attributes to be propagated for the SD-WAN Secure L3 Links are: the attached client routes and the properties of the underlay networks over which the client routes can be carried. The properities of the underlay network may include the following: IPsec-SA information, information needed for NAT transversal with IPsec, and link speeds. Some SD-WAN peers are connected by both trusted VPNs and untrusted public networks. Some SD-WAN peers are connected only by untrusted public networks. For the traffic over untrusted networks, IPsec Security Associations (IPsec SA) must be established and maintained. For the trusted VPNs, IPsec Security associations may not be set-up. If an edge node has network ports behind a NAT, the NAT properties need to be discovered by the authorized SD-WAN peers. Like any VPN networks, the attached client routes belonging to specific SD-WAN VPNs can only be exchanged with the SD-WAN peer nodes authorized to communicate.

Suppose the environment is Figure 1 with the logical SD-WAN Hybrid link topology of Figure 2. These topologies are set-up via configuration with IPsec-SA pre-configured. Suppose that C-PE1 and C-PE2 have 10 pre-shared keys to use on IPsec links. Currently P3 is using IPSec-SA ID-1. C-PE1 wants to receive traffic flowing from C-PE2 over the Hybrid SD-WAN links. Refering to the Figure 2, the C-PE1 routers send customer routes (L3VPN v4 route) to the RR with NextHop set to self (C-PE1), and attaching a Tunnel Encapsulation attribute specifying a Hybrid SD-WAN tunnel to the RR over a secure connection Based on RR policy, the RR forwards routes to other BGP peers that support SD-WAN discovery for that AFI/SAFI (L3VPN v4). Using the Figure 2 example with the L3VPN for v4, RR would forward client routes with TEA of SD-WAN tunnel TLV and Next-Hop of C-PE1. Prior to CE-P2 installing the L3VPN Unicast Routes (1/128), the C-PE2 MUST verify at least one of the underlay connections exist between C-PE1 and C-PE2. Local policy will define which links (or links) the traffic for this route goes over between C-PE1 and C-PE2. The Unsecure Link (P3 Suppose for some reason the L2 link between C-PE1 and C-PE2 has encounter attacks, and the IPsec encryption must now run on links P3 and P4. C-PE1 detects the problem and to change the encryption on P3 and add encryption on P4. C-PE1 and C-PE2 must agree upon the next encryption on the list, and will send the IPsec-SA information in-band via BGP. C-PE1 sends two SD-WAN NLRIs in an BGP Update with a TEA with SD-WAN Hybrid Tunnel TLV with an IPsec-SA-ID TLV with 4 IPsec SA Identifiers (4, 5, 6, 7). The first SD-WAN NLRI contains Port P3's information (Port-local-ID P3, SD-WAN Color Gold (1), and SD-WAN Node-ID CE-PE1). The second SD-WAN NLRI contains Port P4's information (Port-local-id P4, SD-WAN Color Gold (1), and SD-WAN Node-ID CE-PE1). The RR forwards the 2 underlay routes to C-PEs (C-PE1, C-PE2, C-PE3, C-PE4). After receiving C-PE1 update, C-PE2 process the update, and switches the IPsec-SA on Port 3 to IPsec-SA 4, and the IPsec-SA on P4 to IPsec-SA 4. This simple examples show the value of rotating the pre-shared keys. Future IPsec SA can also be set-up, negotiated, or rekeyed in the same manner. The following question may occur to the observant reader: Why is IPsec related information passed on a different AFI/SAFI? Why do you do to support IPsec combined with NATs? Is this the best way to support NATs? The BGP SD-WAN Nodes can attach the TEA with a Hybrid SD-WAN TLV attached to the client routes (AFI/SAFI 1/1, 2/1, 1/128, 2/128) or attached to the SD-WAN NLRI (AFI/SAFI 1/74 or 2/74). The SD-WAN NLRI allows the passing of IPsec information on a unique AFI/SAFI. How implementation prioritize the processing of client routes versus underlay routes (SD-WAN NLRI) is an implementation matter, and out of scope for this document. For NATs, the Extended Port Sub-TLV (see section 3.4) represents the information the first deployment thought it needed to identify the NAT port for a IPsec Security Association. The deployments of this SD-WAN technology found the amount of data gave enough information, but suggested future work might be able to send less information. However, those revisions are outside the scope of this document.

This section describes why this document chose to pass IPSec-SA information via a new BGP AFI/SAFI with a TEA [RFC9012] instead of using traditional IPsec mechanisms. A pure IPsec VPN has IPsec tunnels connecting all edge nodes over public networks. Therefore, it requires stringent authentication and authorization (i.e., IKE Phase 1 [RFC7296]) before other properties of IPsec SA can be exchanged. The IPsec Security Association (SA) between two untrusted nodes typically requires the following configurations and message exchanges: Messages are sent to authenticate with each other. Establishing the IPsec SA requires the following set-up Local key configuration, Setting the Remote Peer address, Information from IKEv2 Proposal directly sent to peer (encryption method, integrity sha512, etc.), and transform set. Running routing protocol within each IPsec SA. If multiple IPsec SAs between two peer nodes are established to achieve load sharing, each IPsec tunnel needs to run its own routing protocol to exchange client routes attached to the edges. Access Lists permit specific Local-IP1, Remote-IP2, and other IPsec parameters. In a BGP-controlled SD-WAN network over hybrid MPLS VPN and public internet underlay networks, all edge nodes and RRs are already connected by private secure paths. The RRs have the policies to manage the authentication of all peer nodes. More importantly, when an edge node needs to establish multiple IPsec tunnels to many edge nodes, all the management information can be multiplexed into the secure management tunnel between RR and the edge node operating as a BGP peer. Therefore, the amount of authentication in a BGP-Controlled SD-WAN network can be significantly reduced. Client VPNs are configured via VRFs, just like the configuration of the existing MPLS VPN. The IPsec equivalent traffic selectors for local and remote routes are achieved by importing/exporting VPN Route Targets. The binding of client routes to IPsec SA is dictated by policies. As a result, the IPsec configuration for a BGP controlled SD-WAN (with mixed MPLS VPN) can be simplified in the following manner: The SD-WAN controller has the authority to authenticate edges and peers so the Remote Peer association is controlled by the SD-WAN Controller (RR). The IKEv2 proposals (including the IPsec Transform set) can be sent directly to peers, or incorporated in a BGP UPDATE. The BGP UPDATE announces the client route reachability through the SDWAN hybrid tunnels. A SDWAN hybrid tunnel combines several other tunnels into a single logical tunnel. The SD-WAN Hybrid tunnel implementations insure that all tunnels within are either running over secure network links or secured by IPsec. Importing/exporting Route Targets under each client VPN (VRF) achieves the traffic selection (or permission) among clients' routes attached to multiple edge nodes. Note: The web of SDWAN hybrid tunnels in a network is denoted in this document as an SD-WAN underlay. BGP passes information about the SDWAN hybrid tunnels between BGP peers by passing an SD-WAN Underlay NLRI paired with the tunnel encapsulation attribute (TEA) with an SDWAN tunnel type SD-WAN-Hybrid TLV. Also, note that with this method there is no need to run multiple routing protocols in each IPsec tunnel.

In SD-WAN Secure L3VPN deployments, SD-WAN Segmentation is a frequently used term which refers to partitioning a network into multiple subnetworks, just like MPLS VPNs. SD-WAN Segmentation is achieved by creating SD-WAN virtual topologies and SD-WAN VPNs. An SD-WAN virtual topology consists of a set of edge nodes and the tunnels (a.k.a. underlay paths) interconnecting those edge nodes. These tunnels forming the underlay paths can be IPsec tunnels, or MPLS VPN tunnels, or other tunnels. Figure 4 (top) illustrates an SD-WAN L3VPN underlay topology and Figure 4 (bottomn) show the same topology as a the virtual topology based on SD-WAN Links. An SD-WAN VPN is configured in the same way as the VRFs of an MPLS VPN. One SD-WAN client VPN can be mapped to multiple SD-WAN virtual topologies. A Route Target is used to differentiate between the SD-WAN VPNs. For example, in the picture below, the Payment-Flow on C-PE2 is only mapped to the virtual topology of C-PEs to/from Payment Gateway, whereas other flows can be mapped to a multipoint-to-multipoint virtual topology. SD-WAN Controller for the SD-WAN Secure L3VPN governs the policies of mapping a client VPN to SD-WAN virtual topologies. Each SD-WAN edge node may need to support multiple VPNs. Route Target is used to differentiate the SD-WAN VPNs. For example, in the picture below, the Payment-Flow on C-PE2 is only mapped to the virtual topology of C-PEs to/from Payment Gateway, whereas other flows can be mapped to a multipoint-to-multipoint virtual topology.

The BGP mechanisms have two functions: A BGP peer supporting SD-WAN re-announces the routes passed by client routers with a next hop self and an BGP attribute indicating the SD-WAN Hybrid Tunnel. Clients routes may be one of the following AFI/SAFIs: Unicast IPv4/IPv6(1/1, 2/1) and L3VPN IPv4/IPv6 (1/128, 2/128). The term "next hop self" means the routers sets the next Hop Address to an address indicating the BGP Peer. The following two BGP Path attributes can pass the Tunnel indication: the Encapsulation Extended Community (Encap-EC) and the Tunnel Encapsulation attribute (TEA). A BGP peer sends the SD-WAN NLRI v4/v6 (AFI/SAFI 1/74 or 2/74) with Next Hop set to Next-Hop-self and an BGP attribute indicating the Hybrid Tunnel. The SD-WAN NLRI identifies the port or (ports) within the Hybrid SD-WAN Tunnel that the BGP peer is sending encapsulation or IPsec-SA related information via the Hybrid SD-WAN TEA. The Hybrid SD-WAN TEA contains the IPsec-SA and optionally NAT information. This section describes the Hybrid SD-WAN tunnel, the SD-WAN NLRIs, the new subTLVs for SD-WAN Tunnel IPSec-SA, subTLVs for Port attributes, the procedures for the client routes, the procedures for underlay routes, error handling, and considerations for managing SD-WAN technologies.

SD-WAN Hybrid Tunnel 25 (IANA assigned) The SD-WAN-Hybrid tunnel identifies a hybrid tunnel that overlays a virtual path over a set links between two BGP Peers comprised of various technologies (e.g. MPLS, L2 direct link, or L3 through Public Internet, and others). These links between the two BGP peers are denoted as underlay links. The underlay links may or may not need encryptions. If some of these underlay links need encryption, the SD-WAN Hybrid Tunnel provides a mechanism to pass this information via BGP. Passage of the IPsec-SA information is optional. Per , the following two BGP attributes that may encode a Tunnel Encapsulation attribute information: the Tunnel Encapsulation Attribute, and the Encapsulation Extended Community (Encap-EC) as a "barebones" tunnel identification. The encoding for the SD-WAN Hybrid tunnel is described for both BGP attributes.

The NextHop Field in the BGP update is the tunnel egress Endpoint, and this should be set to the BGP Peer Address for the SD-WAN Peer.

The Encap-EC format (barebones) of the tunnel encapsulation attribute cannot attach any subTLVs. The valid SubTLV for the TEA form of the Hyrid SD-WAN Tunnel TLV depend whether the TEA is attached to a client route or an underlay route. Table 1 provides a list of valid subTLVs per NLRI type (client or underlay). The valid subTLVs specified [RFC9012] are Color (3) and Tunnel Egress End Point (6). The valid SubTLV specified in this document are: Extended Port Attributes, IPsec SA-ID, IPSec SA Rekey, IPsec Public Key, IPsec SA Proposal SubTLV, and Simplified IPsec SA SubTLV. The validation procedure for the SD-WAN tunnel TLV has the following components: 1) validation of tunnel TLV encoding, 2) Check that SubTLVs are valid for NLRI (see Table 1), and 3) Egress tunnel End Point Check. Prior to installing a route with a SD-WAN tunnel as an active route, the BGP peer installing the route MUST also validate that the SD-WAN tunnel and underlay links are active.

Notes *1 - For client traffic, the validation procedures for the Color subTLV follows the validation procedures of [RFC9012]. Precdence between Color Extended Community (Color-EC) and Color SubTLV is first Color-EC, and then Color SubTLV. *2 - In the future, the SD-WAN TEA could specify more details on Underlay links beyond protocol. However, those future extensions are outside the scope of this document. *3 - Encodings and validation of syntax are defined in section 3.3. Section 3.5 provides content processing and validation procedures for client routes, and section 3.6 provides content processing and validation procedures for underlay routes *4 - Encoding and mechanisms are defined 3.3.6. Section 3.6 provides content processing and validation procedures.

The Edge BGP Peer using BGP SD-WAN discovery sends the hybrid SD-WAN NLRI with the SD-WAN Hybrid tunnel attribute to advertise the detailed properties associated with the public facing WAN ports and IPsec tunnels. The Edge BGP Peer sends this information to its designated RR via the secure connection. Each BGP Update message with a the SD-WAN Underlay NLRI MUST contain a Tunnel Encapsulation Attribute (TEA) for a Hybrid Tunnel type. The TEA can include with SubTLVs for Extended Port attribute (see section 7) or IP Sec information (see section 8). The IPsec information subTLVs include: IPsec-SA-ID, IPsec SA Nonce, IPsec Public Key, IPsec SA Proposal, and Simplified IPsec SA.

A new NLRI SAFI (SD-WAN SAFI=74) is introduced within the MP_REACH_NLRI Path Attribute of [RFC4760] for advertising the detailed properties of the SD-WAN tunnels terminated at the edge node:

where: 2 octet value to define the encoding of the rest of the SD-WAN the NLRI. The SD-WAN Secure Links use case may have many different formats, the route type was assigned two octets. 2 octets of length expressed in bits as defined in [RFC4760]. This document defines the following SD-WAN Route type: For advertising the detailed properties of the SD-WAN tunnels terminated at the edge, where the transport network port can be uniquely identified by a tuple of three values (Port-Local-ID, SD-WAN-Color, SD-WAN-Node-ID). The SD-WAN NLRI Route-Type =1 has the following encoding:

SD-WAN edge node Port identifier, which is locally significant. If the SD-WAN NLRI applies to multiple WAN ports, this field is zero. represents a group of tunnels, which correlate with the Color-Extended-community included in the client routes UPDATE. When a client route can be reached by multiple SD-WAN edges co-located at one site, the SD-WAN-Color can represent a group of tunnels terminated at those SD-WAN edges co-located at the site. If an SD-WAN-Color represents all the tunnels at a site, then the SD-WAN-Color effectively represents the site. The node's IPv4 or IPv6 address. For IPv4 SD-WAN NLRI (1/74) the length is 4. For IPv6 SD-WAN NLRI (1/76), the length is 16. Route Type values outside of 1 are out of scope for this document.

The SD-WAN Node-ID field carries the Tunnel Endpoint. The Tunnel Egress Endpoint in the Tunnel Encapsulation Attribute (TEA) is not used to validate the tunnel indicated by the SD-WAN NLRI. The Next Hop within the UPDATE message MAY be tested by local policy for validity as a BGP Peer source, but the validation of the tunnel endpoint depends solely on the SD-WAN Node-ID.

The BGP Path Attributes for the SD-WAN NLRIs which are required to be supported are: Origin, AS_PATH, Next_HOP, MULTI_EXIT_DISC, LOCAL_PREF (), Originator_ID and Cluster-LIST (), Tunnel Encapsulation Attribute (), Extended Communities (). Per , the Encapsulation Extended Community and the Color Extended Community must be supported. The following optional BGP attributes MAY be attached to an BGP UDPATE with the Hybrid SD-WAN NLRI in the MP-REACH-NLRI or MP_UNREACH_NLRI: Community (), Large Community (), IPv6 Address Specific Extended Community (), AS4_Path and AS4_Aggregator () and These attributes are extensions of required AS_PATH and BGP Community support that may be used to set in local policy evaluations for the Hybrid SD-WAN Underlay. However, the actions of local Policy regarding these BGP attributes is outside scope of this document. All other optional BGP attributes MAY also be attached to the NLRI, but the meaning of these attributes when attached to the NLRI is outside the scope of this document.

This section describes the SubTLVs that pass data regarding IPsec parameters for the Hybrid SD-WAN tunnel. During set-up of the Hybrid SD-WAN tunnels, the IPsec parameters need to be securely passed to set-up secure association. For hybrid SD-WAN tunnels, the IPsec security association for IPsec links may change to different security associations over time. The subTLVs related to IPsec supported by the TEA TLV for the SD-WAN Hybrid Tunnel type are: IPSec-SA-ID, IPsec SA Nounce, IPsec Public Key, IPsec Proposal, and Simplified IPSec SA. The IPSec-SA-ID SubTLV provides a way to indicate the IPsec SA Identifiers (section 3.3.1) for pre-configured security association. The other four SubTLVs provide different ways to pass details regarding IPsec security associations. The IPsec SA Nounce passes Nounce and rekey counters for a Secure Association identified by IPsec SA Identifier (see section 3.3.2). The IPSec Public Key SubTLV passes IPsec Public Key data with a time duration (see section 3.3.3). The IPsec Proposal SubTLV provides Transform attributes and Transform IDs (see section 3.3.4). The Simplified IP SEC SA passes the information that identifies configuration for 2 keys (see section 3.3.5). The NAT-related portion infromation is carried in the Extended Port SubTLV (see section 3.3.6) If any SubTLV is MALFORMED, the implementation MUST follow the procedure in in section 13. For a quick rotation between security associations, the SDWAN NLRI (port-id, color, node) can quickly distribute a switch to a set of new security association using the BGP Update message. In this case, the BGP UPDATE message would like Figure 10

IPsec-SA-ID - Send IPsec SA-IDs 64 (IANA assigned) IPsec-SA-ID Sub-TLV within the Hybrid Underlay Tunnel UPDATE indicates one or more pre-established IPsec SAs by using their identifiers, instead of listing all the detailed attributes of the IPsec SAs. Using an IPsec-SA-ID Sub-TLV not only greatly reduces the size of BGP UPDATE messages, but also allows the pairwise IPsec rekeying process to be performed independently.

where: IPSec-SA-ID Sub-Type (8 bits): 64(IANA Assigned). length (8 bits): Specifies the total length in octets of the value field (not including the Type and Length fields). For the IPSec-SA-ID Sub-Type, the length should be the 2 + 4 *(number of IPsec SA IDs) . Reserved: Reserved for future use. In this version of the document, the Reserved field MUST be set to zero and MUST be ignored upon receipt. Received values MUST be propagated without change. Value field: The value field consists of a sequence of IPsec SA Identifiers, each 4 octets long. As shown in figure above, n IPsec SAs are attached in the IPsec-SA-ID Sub-TLV: IPSec SA Identifier #1: A 4 octet identifier for a pre-established IP security association. IPSec SA Identifier #2: A 4 octet identifier for a pre-established IP security association. IPSec SA Identifier #n: A 4 octet identifier for a pre-established IP security association. A IPsec-SA-ID Sub-TLV is a MALFORMED if the fields do not fit the limits specified above. Per a MALFORMED Sub-TLV is ignored. The procedures for content check for the IPsec-SA-ID Sub-TLV is specified in section 3.4 for client routes, and section 3.5 for underlay routes. The IPsec-SA-ID Sub-TLV does not help validate the Tunnel Egress EndPoint. However, the IPSec-SA provide the security information associated with set of routes sent from a peer which must be correctly handled upon reception of the data.

IPsec SA ReKey Counter - Rekey Counter for a IPsec SA 67 (IANA assigned) The IPsec SA Rekey Counter Sub-TLV provides the rekey counter for a security association (identified by IPsec SA Identifier). The encoding iss shown in the figure below:

where: IPSec-SA-ID Sub-Type (8 bits): 67 (IANA assigned) length (8 bits): Specifies the total length in octets of the value field (not including the Type and Length fields). The total length is variable with the value equal to 16 plus Nonce length. Reserved: Reserved for future use. In this version of the document, the Reserved field MUST be set to zero and MUST be ignored upon receipt. Received values MUST be propagated without change. ID Length (8 bits): indicates the length in octets of SA-Identifer. This length should be 4 octets. Nonce Length (16 bits): indicates the length in octets of the Nonce Data. I Flag: when set to 1, the I-flag indicates that the communication being established is new. when set to 0, it signals that the communication is a continuation of an existing session. Flags (7 bits): Reserved for future use. In this version of the document, the Reserved field MUST be set to zero and MUST be ignored upon receipt. Received values MUST be propagated without change. Rekey Counter (64 bits): the number of key updates or rekeys that have occurred. Each time a key is rotated or replaced, the ReKey Counter is incremented. IPsec SA Identifier (IPSec-SA-ID): a specific IPsec SAs terminated at the edge. Nonce Data: a random or pseudo-random number for preventing replay attacks. Its length is a multiple of 32 bits[RFC7296]. A IPsec SA Rekey Counter Sub-TLV s a MALFORMED if the fields do not fit the limits specified above. Per a MALFORMED SubTLV is ignored. The procedures for content checks for this Sub-tLV are specified in section 3.4 for client routes, and section 3.5 for underlay routes. Please note that: The IPsec-SA-ID may also refer to the values carried in the same TEA in the same Tunnel TLV (type SD-WAN Hybrid) as the IPsec SA Rekey Counter Sub-TLV in either the IPsec Public Key Sub-TLV or the IPsec SA Proposal Sub-TLV. The IPsec SA Rekey Counter, IPsec Public Key, and IPsec SA Proposal Sub-TLVs work together to create security associations. The IPsec-SA-ID may refer to an IPsec SA-ID in another SD-WAN Hybrid Tunnel TLV in the same TEA as the IPsec SA Rekey Counter sub-TLV, The IPsec SA-ID may refer to a preconfigured IPsec SA-ID preconfigured associated with this SD-WAN Hybrid Tunnel Endpoint. The IPsec SA Rekey SubTLV does not help validate the Tunnel Egress EndPoint.

IPsec Public Key - IPsec Public Key information 68 (IANA assigned) The IPsec Public Key Sub-TLV provides the Public Key exchange information and the life span for the Diffie-Hellman Key. The encoding iss shown in the figure below:

where: IPSec-SA-ID Sub-Type (8 bits): 68 (IANA assigned) length (8 bits): Specifies the total length in octets of the value field (not including the Type and Length fields). The total length is variable with the length being 14 + the Key Exchange Data length. Diffie-Hellman Group Num (16-bits): identifies the Diffie-Hellman group used to compute the Key Exchange Data. Details on Diffie-Hellman group numbers can be found in Appendix B of IKEv2 [RFC7296] and [RFC5114]. The Key Exchange data: This refers to a copy of the sender's Diffie-Hellman public value. The length of the Diffie-Hellman public value is defined for MODP groups in [RFC 7296] and for ECP groups in [RFC 5903]. Duration (32 bits): a 4-octet value specifying the life span of the Diffie-Hellman key in seconds. A IPsec Public Key SubTLV is a MALFORMED SubTLV if the fields do not fit the limits specified above. Per a MALFORMED SubTLV is ignored. The procedures for content checks for the IPsec Public is specified in section 3.4 for client routes, and section 3.5 for underlay routes. The IPsec SA Rekey SubTLV not help validate the Tunnel Egress EndPoint.

IPsec SA Proposal - Indicates number of IPsec Transforms 69 (IANA assigned) The IPsec SA Proposal Sub-TLV is to indicate the number of Transform Sub-TLVs. The encoding iss shown in the figure below:

where: IPSec-SA-ID Sub-Type (8 bits): 69 (IANA assigned) length (8 bits): Specifies the total length in octets of the value field (not including the Type and Length fields). The total length is variable with the length being 10 + the Transform attribute length. Transform Attr Length (16 bits): indicates the length of the Transform Attributes field. Transform Type (8 bits): The Transform Type values are defined in Section 3.3.2 of [RFC7296] and [IKEV2IANA]. Only ENCR, INTEG, and ESN values are permitted. Reserved (8 bits): reserved for future use. These bits are ignored upon receipt and set to zero when transmitted. Transform ID (16 bits): An identifer for the transform defined by the associated transform attributes. Transform Attributes: These Sub-SubTLV are defined in section 3.3.5 of [RFC7296]. A IPsec SA Proposal Sub-TLV is a MALFORMED Sub-TLV if the fields do not fit the limits specified above. Per a MALFORMED Sub-TLV is ignored. The procedures for content checks for the IPsec SA Proposal SubTLV is specified in section 3.4 for client routes, and section 3.5 for underlay routes. The IPsec SA Rekey SubTLV not help validate the Tunnel Egress EndPoint.

Simplified IPsec SA - Provides simple key for 2 keys 69 (IANA assigned) For a simple SD-WAN network with edge nodes supporting only a few pre-defined encryption algorithms, a simple IPsec sub-TLV can be used to encode the pre-defined algorithms. The encoding is shown in the figure below:

where: IPSec-SA-ID Sub-Type (8 bits): 70 variable (based on key length) Reserved for future use. These bits SHOULD be set to zero on transmission and MUST be ignored on receipt. Transform = 1 means AH, Transform = 2 means ESP, or Transform = 3 means AH+ESP. All other transform values are outside the scope of this document. Mode = 1 indicates that the Tunnel Mode is used. Mode = 2 indicates that the Transport mode is used. All mode values besides 1 and 2 are outside the scope of this document. AH authentication algorithms supported. The values are specified by [IANA-AH]. Each SD-WAN edge node can support multiple authentication algorithms; send to its peers to negotiate the strongest one. the ESP algorithms supported. Its values are specified by [IANA-ESP]. One SD-WAN edge node can support multiple ESP algorithms and send them to its peers to negotiate the strongest one. The default algorithm is AES-256. When a node supports multiple authentication algorithms, the initial UPDATE needs to include the "Transform Sub-TLV" indicates the count for rekeying. indicates the IPsec public key 1 length IPsec public key 1 indicates the IPsec public key 2 length IPsec public key 2 indicates the Nonce key length IPsec Nonce specifying the security association (SA) life span in seconds. A Simplified IPsec SA Sub-TLVis a MALFORMED Sub-TLV if the fields do not fit the limits specified above. Per a MALFORMED Sub-TLV is ignored. The procedures for content checks for the IPsec SA Proposal SubTLV is specified in section 3.4 for client routes, and section 3.5 for underlay routes. Simplified IPsec SA does not participate in Tunnel End Point Validation.

Extended Port Attribute - Provides information related IPsec and NATs 65 (IANA assigned) The Extended Port Attribute Sub-TLV advertises the properties associated with a public Internet-facing WAN port that might be behind NAT. A SD-WAN edge node can query a STUN Server (Session Traversal of UDP through Network address translation [RFC8489]) to get the NAT properties, including the public IP address and the Public Port number, to pass to its peers. The location of a NAT device can be: Only the initiator is behind a NAT device. Multiple initiators can be behind separate NAT devices. Initiators can also connect to the responder through multiple NAT devices. Only the responder is behind a NAT device. Both the initiator and the responder are behind a NAT device. The initiator's address and/or responder's address can be dynamically assigned by an ISP or when their connection crosses a dynamic NAT device that allocates addresses from a dynamic address pool. As one SD-WAN edge can connect to multiple peers, the pair-wise NAT exchange as IPsec's IKE[RFC7296] is not efficient. In the BGP Controlled SD-WAN, NAT properties for a WAN port are encoded in the Extended Port Attribute sub-TLV. The encoding is shown in the figure below:

where: Extended Port Attribute Type (=65): indicating it is the Extended Port Attribute SubTLV ExtPort Length: the length of the subTLV in octets (variable). Flags: I bit (CPE port address or Inner address scheme): If set to 0, indicate the inner (private) address is IPv4. If set to 1, indicates the inner address is IPv6. O bit (Outer address scheme): If set to 0, indicate the inner (private) address is IPv4. If set to 1, indicates the inner address is IPv6. R bits: reserved for future use. Must be set to 0, and ignored upon reception. NAT Type: the NAT type can be one of the following values: 1: without NAT ; 2: 1-to-1 static NAT; 3: Full Cone; 4: Restricted Cone; 5: Port Restricted Cone; 6: Symmetric; or 7: Unknown (i.e. no response from the STUN server). NAT type values outside of 1-7 are invalid for this SubTLV. Encap-Type: the supported encapsulation types for the port. Encap-Type=1: GRE; Encap-Type=2: VxLAN; Notes: The Encap-Type inside the Extended Port Attribute Sub-TLV is different from the RFC9012's BGP-Tunnel-Encapsulation type. The port can indicate the specific encapsulations, such as: If the IPsec-SA-ID subTLV or the IPsec SA detailed subTLVs (Nonce/publicKey/Proposal) are included in the SD-WAN-Hybrid tunnel, the Encap-Type indicates the encapsulation type within the IPsec payload. If the IPsec SA subTLVs are not included in the SD-WAN-Hybrid Tunnel, the Encap-Type indicates the encapsulation of the payload without IPsec encryption. Encapsulation types outside of GRE and VxLAN are outside of the scope of this specification. Transport Network ID: Central Controller assigns a global unique ID to each transport network. Any value in this octet is valid RD ID: Routing Domain ID, need to be globally unique. Any value in this octet is valid. Some SD-WAN deployment might have multiple levels, zones, or regions that are represented as logical domains. Policies can govern if tunnels can be established across domains. For example, a hub node can establish tunnels with different logical domains but the spoke nodes cannot establish tunnels with nodes in different domains. Local IP: The local (or private) IP address of the WAN port. Local Port: used by Remote SD-WAN edge node for establishing IPsec to this specific port. Public IP: The IP address after the NAT. If NAT is not used, this field is set to all-zeros Public Port: The Port after the NAT. If NAT is not used, this field is set to all-zeros. Extended SubSub-TLV: for carrying additional information about the underlay networks. A IPsec SA Proposal Sub-TLV is a MALFORMED Sub-TLV if the fields do not fit the limits specified above. Per a MALFORMED Sub-TLV is ignored. The procedures for content checks for the IPsec SA Proposal SubTLV is specified in section 3.4 for client routes, and section 3.5 for underlay routes. Simplified IPsec SA does not participate in Tunnel End Point Validation.

One Extended SubSub-TLVs is specified in this document: Underlay Network Transport SubSub-TLV.

Underlay Network Transport - caries WAN port types and speed 66 (IANA Assigned) The Underlay Network Transport SubSub-TLV is an optional SubSub-TLV to carry the WAN port connection types and bandwidth, such as LTE, DSL, Ethernet, and other types. It is only valid for the Tunnel Type Hybrid SD-WAN in the Extended Port Attributes Sub-TLV. The encoding is shown in the figure below:

Where: sub Type=66 always 6 bytes 2 octet of reserved bits. It SHOULD be set to zero on transmission and MUST be ignored on receipt. are listed below as: 1 = Wired 2 = WIFI 3 = LTE 4 = 5G Any value outside of 1-4 is outside the scope of this specification. Port type define as follows: 1 = Ethernet 2 = Fiber Cable 3 = Coax Cable 4 = Cellular Any value outside of values 1-4 are outside the scope of this specification. The port seed is defined as 2 octet value. The values are defined as Gigabit speed. For example, a value of 1 would mean 1 gigabit. The port speed of "0" is not valid. The connection types of equipment and port types will continue to grow with technology change. Future specifications may specify additional connection types or port types. Underlay Network Transport SubSub-TLV is a MALFORMED SubSub-TLV if the fields do not fit the limits specified above. If a MALFORMED SubSubTLV is contained in the Extended Port Attribute SubTLV, then the Extended Port Attribute SubTLV is MALFORMED. Per a MALFORMED Sub-TLV is ignored. The procedures for content checks for the IPsec SA Proposal SubTLV is specified in section 3.4 for client routes, and section 3.5 for underlay routes. Underlay Network Transport SubSub-TLV does NOT contribute to Tunnel End Point Validation.

The Tunnel Encapsulation Attribute for the SD-WAN Hybrid Tunnel Type may be associated with BGP UPDATE messages with NLRI with AFI/SAFI IPv4 Unicast (1/1), IPv6 (2/1), L3VPN v4 Unicast (1/128), and IPv6 L3VPN (2/128). The SD-WAN Secure Links topology is supported by the Unicast IPv4/IPv6 prefixes. The L3VPN topologies support forming the SD-WAN Secure L3VPN described in and MEF ([MEF 70.1][MEF70.2]). Based on [RFC9012], there are two forms a Tunnel Encapsulation Attribute (TEA) can take: "Barebones" using the Encapsulation Extended Community (Encap-EC) and a normal Tunnel Encapsulation form.

The SD-WAN Client routes are sent with a the Encapsulation Extended Community (Encap-EC) BGP attribute that identifies the the Hybrid SD-WAN tunnel type. Per [RFC9012], the Encapsulation Extended Community uses the NextHop Field in the BGP UPDATE as the Tunnel Egress EndPoint. The validation for the Tunnel Egress Endpoint uses the validation in section 6, 8, and 13 applied to the NextHop. A Color Extended Community (Color-EC) or local policy applied to the client route directs the traffic for the client route to across appropriate interface within the Hybrid SD-WAN Tunnel to the Tunnel Egress Endpoint.

The procedures for validating a client route with a TEA does the following: A well-formed Hybrid SD-WAN Tunnel TLV MUST have a Tunnel Engress Endpoint SubTLV. The validation for the Tunnel Egress Endpoint uses the validation in section 6, 8, and 13. An invalid Tunnel Egress Endpoint, cause the Hybrid SD-WAN Tunnel TLV to be invalid, and the TLV is ignored. It MAY also have any of the following SubTLVs: the Color SubTLV defined in , IPsec-SA-ID, IPsec SA Nonce, IPsec Public Key, IPSec SA Proposal, or Simplified IPsec SA) A MALFORMED SubTLV is ignored in the Tunnel TLV is ignored. SubTLV with an unknown type is ignored. Multiple instances of the Tunnel Endpoint SubTLV causes the first one to be used, and the subsequent instances to be ignored. Multiple instances of the IPsec Public Key, IPsec SA Proposal, and Simplified IPsec SA cause the first instance to be used, and subsequent instances to be ignored. A Hybrid SD-WAN tunnel TLV may have multiple instances of the IPsec-SA-ID if the IPsec SA Identifiers are unique. If all the IPsec SA Identifies are not unique, the second SubTLV is ignored and not propagated. The Tunnel Egress Endpoint MUST link to the remote end of one of the underlay links to be used. This validation adheres to the Tunnel Egress Endpoint validation. The tunnel link may be active or inactive. Local policy is run to validate routes. The next hop must be be reachable via the tunnel."

A single SD-WAN client route may be attached to multiple SD-WAN Hybrid tunnels. An Update with an SD-WAN client route may express these tunnels as an Encap-EC or a TEA. Each of these tunnel descriptions is treated as a unique Hybrid SD-WAN tunnel with a unique Egress Endpoint. Local Policy on the BGP Peer determines which tunnel the client data traffic will use.

An SD-WAN VPN ID is the same as a client VPN ID in a BGP controlled SD-WAN network. The Route Target Extended Community should be included in a Client Route UPDATE message to differentiate the client routes from routes belonging to other VPNs. Route Target value is taken as the VPN ID (for 1/1 and 2/1). For 1/128 and 2/128, the RD from the NLRI identifies the VPN ID. For EVPN, picking up the VPN-ID from EVPN SAFI.

SD-WAN edge node can be reached by either an MPLS path or an IPsec path within the hybrid SD-WAN tunnel. If client packets are sent via a secure MPLS network within the Hybrid SD-WAN tunnel, then the data packets will have MPLS headers with the MPLS Labels based on the scheme specified by [RFC8277]. It is assumed the secure MPLS network assures the security outer MPLS Label header. If the packets are sent via a link with IPsec outer encryption across a public network, the payload is still encrypted with GRE or VXLAN encryption. For GRE Encapsulation within an IPsec tunnel, the GRE key field can be used to carry the SD-WAN VPN ID. For network virtual overlay (VxLAN, GENEVE, etc.) encapsulation within the IPsec tunnel, the Virtual Network Identifier (VNI) field is used to carry the SD-WAN VPN ID.

The Hybrid SD-WAN NLRI MUST be accompanied with the TEA, and MUST NOT be accompanied by an Encapsulation Extended Community.

An underlay routes contains Hybrid SD-WAN NLRIs with TEA attached. The procedures for processing underlay routes follows the following steps: An Hybrid SD-WAN Tunnel TLV is well-formed using only SubTLVs valid for association with the underlay Route. A well-formed SD-WAN Hybrid Tunnel TLV MUST contain a valid Tunnel Egress Endpoint subTLV to be a valid SD-WAN Hybrid TLV. provides validation guidelines for the Tunnel Egress Endpoint in sections 13. The SD-WAN Hybrid Tunnel TLV MAY contain the following subTLVs: Tunnel Egress, IPsec-SA-ID, Extended Port SubTLV, IPsec Nonce, IPsec Public Key, IPsec SA Proposal, and Simplified IPsec SA. Following intent, a MALFORMED SubTLV is ignored, and a SubTLV with an unknown type is ignored. Multiple instances of the Tunnel Endpoint, IPsec Public Key, IPsec SA Proposal, and Simplified IPsec SA cause only the first one to be used. Subsequent subTLV ware ignored and not propagated. The IPsec-SA-ID subTLVs may have multiple instances of the subTLV if the IPsec SA Identifiers are unique, but if the IPsec SA Identifiers are not unique the second subTLV is ignored and not propagated. If multiple Extended Port SubTLVs exist, the TLVs must be validated in step 4. The Tunnel Egress Endpoint MUST link to remote end point of one of the underlay links from the router receiving the link. If a single Extended Port SubTLV exist, then validate that the port information provides needed information to establish a connection to the remote port. A SD-WAN edge node MAY utilize local policy, local configuration, and the information from the SubTLV. The exact mechanism of local port validation is outside the scope of this document. If the Extended Port SubTLV is invalid, the SD-WAN tunnel TLV must be considered to be invalid. The typed NLRI in the SD-WAN Underlay MUST be Well-Formed having the format specified in section 3.2.1. A MALFORMED NLRI must cause the NLRI to be discarded. An implementation MAY log an error for a MALFORMED NLRI. The local policy configuration in the BGP peer receiving this NLRI MUST determine the validity of the route based on policy. The IP address specified in the Next Hop field must be reachable by the Tunnels.

Section 3.2.1 specifies that Route Type 1 has a tuple of (Port-Local-ID, SD-WAN-Color, SD-WAN-Node-ID). Port-Local-ID may be zero if the NLRI applies to multiple ports. The BGP Peer receiving the NLRI must have pre-configured inbound filters to set the preference for the SD-WAN NLRI tuple. Since a Port-Local-ID value of zero indicates the NLRI applies to multiple ports, it is possible to have the following NLRI within a packet (or received in multiple packets): Port-Local-ID (0), SD-WAN-Color (10), SD-WAN-Node-ID (2.2.2.2), Port-Local-ID (0), SD-WAN-Color (20), SD-WAN-Node-ID (2.2.2.2), and Port-Local-ID (0), SD-WAN-Color (30), SD-WAN-Node-ID (2.2.2.2). These NLRI may simply indicate that there are three groups of tunnels for SD-WAN-Node-ID (2.2.2.2) assigned three colors. For example, these tunnels could represent three types of gold, silver and bronze network service.

An underlay route (SD-WAN NLRI) may only attach to one Hybrid SD-WAN Tunnel. If there are more than one Hybrid SD-WAN Tunnel TLV within a single TEA, the first is processed and the subsequent Hybrid SD-WAN Tunnel TLVs are ignored.

The Error handling for SD-WAN VPN support has two components: error handling for Tunnel Encapsulation signaling (Encap-EC and TEA) and the SD-WAN NLRI. An SD-WAN NLRI, a Tunnel Encapsulation attribute MUST always accompany the SD-WAN NLRI. The previous sections (3.4 and 3.5) provide the normal procedures for handling client routes and undelay routes.

The error handling for the tunnel encapsulation signaling (Encap-EC and TEA) adheres to the error handling and validation specified by [RFC9012]. The Tunnel encapsulation signaled with the client routes indicates the Egress endpoint via Next Hop in the Encap-EC or the TEA SubTLV for Tunnel Egress Endpoints. As indicated in the procedure in sections 3.4.2 and 3.5.2, the SD-WAN Hybrid tunnel follows the validation section 6, 8, and 13 from [RFC9012]. The SD-WAN client routes associates some of the NLRIs that [RFC9012] associates with the Encap-EC and the TEA using the validation specified in [RFC9012] in sections 6, 8, and 13. When the SD-WAN Hybrid Tunnel is associated with the SD-WAN NLRI, and all RFC9012 validation rules in section 6, 8, and 13 are extended to apply to the SD-WAN NLRI. [RFC9012] contains the necessary detail to specify validation for the new SubTLVs present for the SD-WAN Tunnel type. However, to aid users of this document the following recap of validation of [RFC9012] is provided below. The validation from section 13 of [RFC9012] includes: Invalid tunnel type must be treat if the TLV was not present. A malformed subTLVs must be treated as an unrecognized subTLV except for Tunnel Egress Endpoint. If Tunnel Egress Endpoint is malformed, the entire TLV must be ignored. Multiple incidents of Tunnel Egress Endpoint SubTLV cause the first incident of these subTLVs to be utilized. Subsequent TLVs after the first one per type are ignored (per RFC9012), but propogated. If a subTLV is meaningless for a tunnel type, the subTLV is ignored, but the subTLV is not considered malformed or removed from the Tunnel Attribute propagated with the NLRI. For SD-WAN client routes with a TEA with a SD-WAN Hybrid Tunnel type TLV, the IPSec SubTLVs (IPsec SA-ID, IPSec nonce, IPSec Public Key, IPsec Proposal, and Simplified IPsec SA) are meaningful, but may be rarely sent. Incorrect fields within any of these 5 TLVs. Per [RFC9012], a malformed subTLV is treated as an unrecognized subTLV. For SD-WAN NLRI underlay routes, the the Extended Port subTLV and the IPSec SubTLVs (IPsec SA-ID, IPSec nonce, IPSec Public Key, IPsec Proposal, and Simplified IPsec SA) are valid and meaningful. Incorrect fields within any of these 6 TLVs or subSubTLVs within the TLVs should cause the subTLV to be treated as malformed SubTLV. Per [RFC9012], a malformed subTLV is treated as an unrecognized subTLV. If multiple instancs of the IPsec nonce, IPsec Public Key, IPsec Proposal, and Simplified IPsec are received within a SD-WAN Tunnel TLV , only the first is processed. The second instance is ignored and not propagated. The IPsec SA-ID may have multiple copies, but the IPsec SA Identifiers sent in the second SubTLV must be different than any in the first IPsec-SA ID SubTLV. If multiple instances of the Extended Port subTLV are received, the local policy must determine which is to be used.

The SD-WAN NLRI [AFI 1/SAFI = 74] utilizes a route type field to describe the format of the NLRI. This specification only allows an NLRI with a type value of 1. An NLRI with a type of field of another value is ignored and not processed. The implementation MAY log an error upon a reception of an type value outside of Route Type 1. Error handling for the SD-WAN NLRI also adheres to the BGP Update error handling specified in [RFC7606]. The local policy configuration in the BGP peer receiving this NLRI must determine the validity of the route based on policy. Local configuration and policy must be carefully constrain the SD-WAN-NLRI, tunnels, and IPsec security associations in to create a "walled garden". In the future, other proposals for a SD-WAN NLRI may specify a different route type. Those proposals must specify the following: validation for new Route Type in the SD-WAN-NLRI, and how the new Route Type interacts with the Route Type 1.

The SD-WAN NLRI (AFI=1/SAFI=74) must be paired with Tunnel Encapsulation attribute with a tunnel TLV for tunnel type SD-WAN-Hybrid. If the SD-WAN NLRI exist in an BGP UPDATE without a Tunnel Encapsulation Attribute (TEA) with a tunnel TLV for tunnel type SD-WAN-Hybrid, the NLRI is considered malformed and Treat-as-withdraw approach (per RFC7606). The SD-WAN NLRI should not be paired with Encapsulation Extended Community. If an SD-WAN NLRI is paried Encapsulation Extended Community rather than a Tunnel Encapsulation Attribute, the SD-WAN NLRI is considered malformed and Treat-as-withdraw approach (per ) should be used.

Unlike MPLS VPN whose PE nodes are all controlled by the network operators, SD-WAN edge nodes can be installed anywhere, in shopping malls, in 3rd party Cloud DCs, etc. It is very important to ensure that client routes advertisement from an SD-WAN edge node are legitimate. The RR needs to ensure the SD-WAN Hybrid Tunnels and routes run over the appropriate Security associations.

It is critical that the Hybrid SD-WAN Tunnel have correctly forward traffic based on the local policy on the client routes, the tunnel egress and tunnel ingress, and the security association. The RR reflector and the BGP peer must check that the client routes, tunnel egress, tunnel ingress, and security associations align with expected values for a tunnel.

Each BGP peer (e.g. a C-PE) advertises a SD-WAN SAFI Underlay NLRI to the other BGP peers via a BGP Route Reflector to establish pairwise IPsec Security Associations (SA) between itself and other remote BGP Peers. During the SD-WAN SAFI NLRI advertisement, the BGP Peer originating may pass information about security association in one of three forms: an identifier for a pre-configured and established IPsec Security Association, a simplified set of security parameters for setting up a IPsec Security association (Transform, IPsec Mode, AH and ESP Algorithms, rekey counter, 2 public keys, nonce, and duration of security association), or a flexible set of security parameters where Nonce, Public Key, and SA Proposal are uniquely specified. For existing IPsec Security associations, the receiving BGP peer can simply utilize one of these existing security associations to pass data. If multiple IPsec associations are pre-configured, the local policy on the SD-WAN Edge Node may help select which security association is chosen for the SD-WAN Hybrid Tunnel. If the receiving and originating BGP peer engage in a set-up for the IPsec security associations for the link within the SD-WAN Hybrid tunnel, IPsec mechansims require that there are matching IPsec transforms. Without common IPsec transforms, the IPsec set-up process cannot operate.

The TEA passes in the Tunnel TLV for the SD-WAN Hybrid Tunnel these three sets of information in the following subTLVs: passes the previous configured (pre-configured or generated) IPsec SA identifiers. specifies a simplified set of information upon which to set-up the IPsec security associations for the tunnel. IPsec SA Rekey Counter SubTLV, IPsec Public Key SubTLV, and a IPSec Proposal SubTLV. Configuration on the local node uses this information and any information in the underlay to create security associations. The BGP Peer's need to send the IPSec SA attributes received on the SD-WAN NLRI in the TEA between the local and remote WAN ports. If there is a match on the SA Attributes between the two ports, the IPSec Tunnel is established. If there is a mismtach on the SA Attributes, no IPsec Tunnel is established. The C-PE devices do not try to negotiate the base IPSec-SA parameters between the local and the remote ports in the case of simple IPSec SA exchange or the Transform sets between local and remote ports. If there is a mismatch in the IPsec SA, then no IPsec Tunnel is created. If there is a mismatch on the Transform sets in the case of full-set of IPSec SA Sub-TLVs, no tunnel is created.

This section provides one example of how IPsec Security associations are created over the SD-WAN Hybrid tunnel. Figure 1 in Section 3 shows an establish an IPsec Tunnel being created between C-PE1 and C-PE2 WAN Ports A2 and B2 (A2: 192.10.0.10 - B2:192.0.0.1). To create this tunnel C-PE1 needs to advertise the following attributes for establishing the IPsec SA: NextHop: 192.10.0.10 SD-WAN Node ID: 1.1.1.1 SD-WAN-Site-ID: 15.0.0.2 Tunnel Encap Attr (Type=SD-WAN) - Extended Port Attribute Sub-TLV containing Transport SubSubTLV - with information on ISP3. IPSec information for detailed information about the ISP3 IPsec SA Rekey Counter Sub-TLV, IPsec SA Public Key Sub-TLV, Proposal Sub-TLV (type = ENCR, transform ID = 1) type: ENCR Transform ID: 1 Tranform attributes = trans 1 [from RFC7296] C-PE2 needs to advertise the following attributes for establishing IPsec SA: 192.0.0.1 2.2.2.2 1500 Extended Port Attribute SubTLV Transport SubSubTLV - with information on ISP3. IPsec SA Rekey Counter Sub-TLV, IPsec SA Public Key Sub-TLV, IPSec Proposal Sub-TLV with transform type: ENCR Transform ID = 1 Transform attributes = trans 2 As there is no matching transform between the WAN ports A2 and B2 in C-PE1 and C-PE2, respectively, no IPsec Tunnel will be established.

The document describes the encoding for SD-WAN edge nodes to advertise its properties to their peers to its RR, which propagates to the intended peers via a secure link across an untrusted network. The secure propagation is achieved by secure channels, such as TLS, SSL, or IPsec, between the SD-WAN edge nodes and the local controller RR. It is assumed that the SD-WAN edges communication will result in aligned IPsec Attributes. Please see section 4.2 for a discussion of what happens if the IPsec Attributes are mismatched. SD-WAN edge nodes MUST have a secure transport connection to the RR. This secure BGP connection can be established as BGP using TCP over IPsec, SSL or TLS. In a walled garden SD-WAN deployment where all SD-WAN edges and the central controller are under one administrative control and the network operates within a closed environment, the threat model is primarily on internal threats, misconfigurations, and localized physical risks. Unauthorized physical access to SD-WAN edge devices in remote locations is a concern. Such access might allow attackers to compromise the edge devices and potentially manipulate the advertised Client prefixes with VPN IDs (or Route Targets) that do not belong to them. This can lead to unauthorized data interception and traffic redirection. Ensuring secure communication between SD-WAN edge nodes and the central controller within a walled garden deployment is crucial. It is essential to utilize secure communication channels such as TLS, SSL or TCP over IPsec for all communications between edge nodes and the controller. Therefore, it is necessary to ensure physical security controls are in place at remote locations, including locks, surveillance, and access controls. Additionally, the RR needs to verify the BGP advertisements from each SD-WAN edge to ensure in the SD-WAN Secure L3VPN that their advertised VPN IDs (and Route Targets) are truly theirs. In addition, local BGP policy should careful set-up access lists for the routes received and propagated. These verifications help prevent unauthorized advertisement of prefixes and ensures the integrity of the routing information within the SD-WAN environment. This document does not specify a SD-WAN deployment outside of the above walled garden described above. Such a deployment is outside the scope of this document.

IANA has assigned SAFI = 74 as the Hybrid (SD-WAN) SAFI.

IANA is requested to assign a type from the BGP Tunnel Encapsulation Attribute Tunnel Types as follows [RFC8126]:

IANA is requested to assign the following sub-Types in the BGP Tunnel Encapsulation Attribute Sub-TLVs registry:

IANA is requested to assign a type from the BGP Tunnel Encapsulation Attribute Tunnel Types as follows [RFC8126]:

L. Dunbar, A Malis, C. Jacquenet, M. Toy and K. Majumdar L. Dunbar, A Sajassi, J Drake, and B. Najem MEF MEF IANA IANA

Acknowledgements to Wang Haibo, Shunwan Zhuang, Hao Weiguo, and ShengCheng for implementation contribution. Many thanks to Yoav Nir, Graham Bartlett, Jim Guichard, John Scudder, and Donald Eastlake for their review and suggestions.

Contributors Below is a list of other contributing authors: Verizon

USA gyan.s.mishra@verizon.com

, Huawei

Beijing China zhuangshunwan@huawei.com

, Huawei

Beijing China shengcheng@huawei.com

, and Futurewei

USA d3e3e3@gmail.com

.