<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?Pub Inc?>
<?rfc toc="yes"?>
<?rfc tocompact="yes"?>
<?rfc tocdepth="3"?>
<?rfc tocindent="yes"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes"?>
<?rfc comments="yes"?>
<?rfc inline="yes"?>
<?rfc compact="yes"?>
<?rfc subcompact="no"?>
<rfc category="std" docName="draft-ietf-idr-sdwan-edge-discovery-12"
     ipr="trust200902">
  <front>


    <title abbrev="SDWAN Edge Discovery">BGP UPDATE for SD-WAN Edge Discovery</title>

    <author fullname="Linda Dunbar" initials="L. " surname="Dunbar">
      <organization>Futurewei</organization>

      <address>
		<postal>
          <city>Dallas, TX</city>
          <country>USA</country>
		</postal>
        <email>ldunbar@futurewei.com</email>
      </address>
    </author>

    <author fullname="Kausik Majumdar" initials="K." surname="Majumdar">
      <organization>Microsoft Azure</organization>

      <address>
        <postal>
          <street/>

          <city>California</city>

          <country>USA</country>
        </postal>

        <email>kmajumdar@microsoft.com</email>
      </address>
    </author>

    <author fullname="Susan Hares" initials="S. " surname="Hares">
      <organization>Hickory Hill Consulting</organization>

      <address>
        <postal>
          <street></street>

          <city> </city>
          <country>USA</country>
        </postal>
        <email>shares@ndzh.com</email>
      </address>
    </author>

    <author fullname="Robert Raszuk" initials="R. " surname="Raszuk">
      <organization>Arrcus</organization>

      <address>
        <postal>
          <street></street>

          <code></code>

          <country>USA</country>
        </postal>

        <email>robert@raszuk.net</email>
      </address>
    </author>

    <author fullname="Venkit Kasiviswanathan" initials="V" surname="Kasiviswanathan">
      <organization>Arista</organization>

      <address>
        <postal>
          <street/>

          <city></city>

          <country>USA</country>
        </postal>

        <email>venkit@arista.com</email>
      </address>
    </author>


    <date year="2023"/>

    <abstract>
     <t>The document describes the encoding of BGP UPDATE messages for the 
	 SD-WAN edge node property discovery.</t> 
	<t>In the context of this document, BGP Route Reflector (RR) is the component of the 
	SD-WAN Controller that receives the BGP UPDATE from SD-WAN edges 
	and in turns propagates the information to the intended peers 
	that are authorized to communicate via the SD-WAN overlay network.</t>

    </abstract>

    <note title="Requirements Language">
      <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
      "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
      document are to be interpreted as described in <xref
      target="RFC2119"></xref> <xref target="RFC8174"></xref>
      when, and only when, they appear in all
      capitals, as shown here.</t>
    </note>
  </front>

  <middle>
    <section title="Introduction">
      

      <t><xref target="SD-WAN-BGP-USAGE"/>
      illustrates how BGP [RFC4271] is used as a control plane 
	  for a SD-WAN network. SD-WAN network refers to a policy-driven network 
	  over multiple heterogeneous underlay networks to get better 
	  WAN bandwidth management, visibility, and control. </t>
	  <t>The document describes BGP UPDATE messages for an SD-WAN edge 
	  node to advertise its properties to its RR which then propagates 
	  that information to the authorized peers. 
	  </t>
   </section>

   <section title="Conventions used in this document">
     <t>The following acronyms and terms are used in this document: </t>
	 <list
          style="hanging">
          <t hangText="Cloud DC:">Off-Premises Data Centers that usually host applications and workload owned by different organizations or tenants.</t>
		  <t hangText="Controller:">Used interchangeably with SD-WAN controller to manage SD-WAN overlay path creation/deletion and monitor the path conditions between sites</t>
		  <t hangText="CPE:">Customer (Edge) Premises Equipment</t>
		  <t hangText="CPE-Based VPN:">Virtual Private Secure network formed among CPEs. This is to differentiate such VPNs from most commonly used PE-based VPNs discussed in [RFC4364].</t>
		  <t hangText="MP-NLRI:">Multi-Protocol Network Layer Reachability Information [MP_REACH_NLRI] Path Attribute defined in [RFC4760]</t>
		  <t hangText="RR:">Route Reflector</t>
		  <t hangText="SD-WAN:">An overlay connectivity service that optimizes transport of IP Packets over one or more Underlay Connectivity Services by recognizing applications (Application Flows) and determining forwarding behavior by applying Policies to them. [MEF-70.1]</t>
		   <t hangText="SD-WAN Endpoint:">can be the SD-WAN edge node address, a WAN port address (logical or physical) of a SD-WAN edge node, or a client port address.</t>
		  <t hangText="VPN:">Virtual Private Network</t>
		  <t hangText="VRF:">VPN Routing and Forwarding instance</t>
		  <t hangText="WAN:">Wide Area Network</t>

		  
	</list> 
	<t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC8174] when, and only when, they appear in all capitals, as shown here.
	</t>
   </section>
   
	<section title="Framework of SD-WAN Edge Discovery">
	<section title="The Objectives of SD-WAN Edge Discovery">
	<t>
	The objectives of SD-WAN edge discovery for an SD-WAN edge node 
	are: to discover its authorized peers and their associated properties 
	in order to establish secure overlay tunnels [Net2Cloud]. The attributes to be propagated includes: </t> 
	<list style="symbol">
	<t>the SD-WAN (client) VPNs information, </t>
	<t>the attached routes under the SD-WAN VPNs, and </t>
	<t>the properties of the underlay networks over 
	 which the client routes can be carried, and potentially more.</t>
	</list>
	<t>Some SD-WAN peers are connected by both trusted VPNs and untrusted public networks. 
	Some SD-WAN peers are connected only by untrusted public networks. 
	For the traffic over untrusted networks, IPsec Security Associations (IPsec SA) 
	must be established and maintained. If an edge node has network ports behind a NAT, 
	the NAT properties need to be discovered by the authorized SD-WAN peers.</t>
	
	<t>Like any VPN networks, the attached client routes belonging 
	to specific SD-WAN VPNs can only be exchanged with the SD-WAN 
	peer nodes authorized to communicate.</t>

	</section>
	<section title="Comparing with Pure IPsec VPN">
	<t> A pure IPsec VPN has IPsec tunnels connecting all edge nodes over public networks. 
	Therefore, it requires stringent authentication and authorization (i.e., IKE Phase 1) 
	before other properties of IPsec SA can be exchanged. The IPsec Security Association (SA) 
	between two untrusted nodes typically requires the following configurations and message exchanges:
	</t>
	<t>
	<list>
	<t>
	<list style="hanging">
	<t hangText="IPsec IKEV2:">messages sent to authenticate with each other. </t> 
	<t hangText="Establish IPsec SA">: requires the following set-up  
		<list style="symbol">
		<t>Local key configuration, </t>
		<t>Setting the Remote Peer address (such as 192.10.0.10 - 172.0.01 </t>
		<t>Information from IKEv2 Proposal directly sent to peer: (Encryption method, Integrity sha512, etc.)</t>
		<t>Transform set. </t>
		</list>	
	 </t>
	<t hangText="Attached client prefixes discovery:">which is achieved by:
		<list style="symbol">
		<t>Running routing protocol within each IPsec SA. </t>
		<t>If multiple IPsec SAs between two peer nodes are established to achieve load sharing, each IPsec tunnel needs to run its own routing protocol to exchange client routes attached to the edges.</t>
		</list>	
		</t>
	<t hangText="Set-up Access List or Traffic Selector:">such as Permit Local-IP1, Remote-IP2, and other parameters.</t>
	</list>
	</t>
	</list>
	</t>
	<t>In a BGP-controlled SD-WAN network over hybrid MPLS VPN and public internet underlay networks, 
	all edge nodes and RRs are already connected by private secure paths. The RRs have the policies 
	to manage the authentication of all peer nodes. More importantly, when an edge node needs to 
	establish multiple IPsec tunnels to many edge nodes, all the management information can 
	be multiplexed into the secure management tunnel between RR and the edge node. 
	Therefore, the amount of authentication in a BGP-Controlled SD-WAN network can be significantly reduced.</t> 
	
	<t>Client VPNs are configured via VRFs, just like the configuration of the existing MPLS VPN. 
	The IPsec equivalent traffic selectors for local and remote routes are achieved by 
	importing/exporting VPN Route Targets. The binding of client routes to IPsec SA is dictated by policies. 
	As a result, the IPsec configuration for a BGP controlled SD-WAN (with mixed MPLS VPN) can be simplified
	in the following manner:</t>
	<t>
	<list style="symbol">
	<t>The SD-WAN controller has the authority to authenticate edges and peers 
	so the Remote Peer association is controlled by the SD-WAN Controller (RR).  </t>
	<t>The IKEv2 proposals (including the IPsec Transform set) 
	   can be sent directly to peers, or incorporated in a BGP UPDATE. </t>		
	<t>The BGP UPDATE announces the client route reachability for all permitted parallel tunnels/paths.
    </t>
	<t>Importing/exporting Route Targets under each client VPN (VRF) achieves the traffic selection 
	(or permission) among clients' routes attached to multiple edge nodes.</t>
	</list>
	</t>
	<t>Note that with this method there is no need to run multiple routing protocols in each IPsec tunnel.</t>	
	</section>
	<section title="Client Route UPDATE and SD-WAN Tunnel UPDATE">
	<t>As described in [SD-WAN-BGP-USAGE], two separate BGP UPDATE messages are used for SD-WAN Edge Discovery:</t>
	<list style="hanging">
	<t hangText="-">Client routes BGP UPDATE:</t>
	<t>This UPDATE is precisely the same as the BGP VPN client route UPDATE. It uses the Encapsulation Extended Community and the Color Extended Community to link with the SD-WAN Tunnels UPDATE Message as specified in section 8 of [RFC9012].</t>
	<t>A new Tunnel Type (SD-WAN-Hybrid) is added and used by the Encapsulation Extended Community or the Tunnel-Encap Path Attribute [RFC9012] to indicate mixed underlay networks.</t>
	<t hangText="-">SD-WAN UPDATE:</t>
	<t>This UPDATE is for an edge node to advertise the properties of directly attached underlay networks, including the NAT information, pre-configured IPsec SA identifiers, and/or the underlay network specific information. This UPDATE can also include the detailed IPsec SA attributes, such as keys, nonce, encryption algorithms, etc.</t>
	</list>
	<t>In the following figure, four overlay paths between C-PE1 and C-PE2 are established for illustration purpose. More overlay paths are possible. One physical port on C-PE2 can terminate multiple overlay paths from different ports on C-PE1. </t>
	<list style="hanging">
	<t hangText="a)">MPLS-in-GRE path;</t>
	<t hangText="b)">node-based IPsec tunnel [2.2.2.2 - 1.1.1.1]. As C-PE2 has two public internet facing WAN ports, either of those two WAN port IP addresses can be the outer destination address of the IPsec encapsulated data packets; </t>
	<t hangText="c)">port-based IPsec tunnel [192.0.0.1 - 192.10.0.10]; and</t>
	<t hangText="d)">port-based IPsec tunnel [172.0.0.1 - 160.0.0.1]. </t>
	</list>
	<figure
       anchor="Hybrid SD-WAN"
       title="Hybrid SD-WAN">
       <artwork><![CDATA[ 
                                  +---+
                   +--------------|RR |----------+
                  /  Untrusted    +-+-+           \
                 /                                 \
                /                                   \
        +---------+  MPLS Path                      +-------+   
11.1.1.x| C-PE1   A1-------------------------------B1 C-PE2 |10.1.1.x
        |         |                                 |       |
21.1.1.x|         A2(192.10.0.10)------( 192.0.0.1)B2       |20.1.1.x  
        |         |                                 |       |
        | Addr    A3(160.0.0.1) --------(170.0.0.1)B3 Addr  |   
        | 1.1.1.1 |                                 |2.2.2.2|
        +---------+                                 +-------+ 
]]></artwork>
	</figure> 
	<t>C-PE2 advertises the attached client routes as below:
	<list style="symbol">
	<t>Client Route UPDATE:
	<list>
	<t>Extended community: RT for SD-WAN VPN 1 </t>
	<t>NLRI: AFI=IPv4/IPv6 and SAFI = VPN </t>
	<t>    Prefix: 10.1.1.x; 20.1.1.x </t>
	<t>    NextHop: 2.2.2.2 (C-PE2) </t>
	<t>Encapsulation Extended Community: tunnel-type=SD-WAN-Hybrid</t>
	<t>Color Extended Community: Site-identifier</t>
	</list>
	</t>
	<t>SD-WAN UPDATE:
	<list>
	<t>C-PE2 can use the following Update messages to 
	advertise the properties of Internet facing ports 192.0.0.1 and 170.0.0.1, 
	and their associated IPsec SA related parameters.</t> 
	<t>Update #1 for the properties associated with the WAN port 192.0.0.1, such as the NAT properties, the underlay network properties, etc.  (Details in Section 9.1).</t>
	<t>Update #2 for the properties associated with the WAN port 170.0.0.1 associated properties. (Details in Section 9.1).</t>
	<t>Update #3 for IPsec parameters associated with IPsec tunnel terminated at the Node level (2.2.2.2), such as the supported encryption methods, public keys, etc. (Details in Section 9.2).</t>
	</list>
	</t>
	</list>
	</t>
	</section>
	<section title="Edge Node Discovery ">
	<t>The basic scheme of SD-WAN edge node discovery using BGP consists of the following:
	<list style="symbol">
		<t>Secure connection to a SD-WAN controller (in this context the RR ):
		<list> 
		<t>For an SD-WAN edge with both MPLS and IPsec paths, the edge node should already have a secure connection to 
		its controller (the RR in this context). For an SD-WAN edge that is only accessible via Internet, 
		the SD-WAN edge, upon power-up, establishes a secure tunnel (such as TLS or SSL) 
		with the SD-WAN central controller whose address is preconfigured on the edge node. 
		The central controller informs the edge node of its local RR. The edge node then establishes a 
		transport layer secure session with the RR (such as TLS or SSL).</t>
		</list>
		</t>
		<t>The Edge node will advertise its own properties to its designated RR via the secure connection.</t>
		<t>The RR propagates the received information to the authorized peers. </t>
		<t>The authorized peers can establish the secure data channels (IPsec) and exchange more information 
		among each other. </t>
	</list>
	</t>
	<t>For an SD-WAN deployment with multiple RRs, it is assumed that there are secure connections among those RRs. 
	How secure connections are established among those RRs is out of the scope of this document. 
	The existing BGP UPDATE propagation mechanisms control the edge properties propagation among the RRs.</t>
	<t>For some environments where the communication to RR is highly secured, [RFC9016] 
	IKE-less can be deployed to simplify IPsec SA establishment among edge nodes.</t>
	</section>
	</section>
<section title="Constrained propagation of BGP UPDATE ">
<section title="SD-WAN Segmentation, Virtual Topology and Client VPN">
	<t>In SD-WAN deployment, SD-WAN Segmentation is a frequently used term, referring to partitioning a network into multiple subnetworks, just like MPLS VPNs. SD-WAN Segmentation is achieved by creating SD-WAN virtual topologies and SD-WAN VPNs. An SD-WAN virtual topology consists of a set of edge nodes and the tunnels (a.k.a. underlay paths), including both IPsec tunnels and/or MPLS VPN tunnels, interconnecting those edge nodes.</t> 
	<t>An SD-WAN VPN is configured in the same way as the VRFs of an MPLS VPN. One SD-WAN client VPN can be mapped to multiple SD-WAN virtual topologies. SD-WAN Controller governs the policies of mapping a client VPN to SD-WAN virtual topologies.</t>
	<t>Each SD-WAN edge node may need to support multiple VPNs. Route Target is used to differentiate the SD-WAN VPNs. For example, in the picture below, the Payment-Flow on C-PE2 is only mapped to the virtual topology of C-PEs to/from Payment Gateway, whereas other flows can be mapped to a multipoint-to-multipoint virtual topology.</t>
	<figure
       anchor="SD-WAN Virtual Topology and VPN"
       title="SD-WAN Virtual Topology and VPN">
       <artwork><![CDATA[ 
                                  +---+
                   +--------------|RR |----------+
                  /  Untrusted    +-+-+           \
                 /                                 \
                /                                   \
        +---------+  MPLS Path                      +-------+   
11.1.1.x| C-PE1   A1-------------------------------B1 C-PE2 |10.1.1.x
        |         |                                 |       |
21.1.1.x|         A2(192.10.0.10)------( 192.0.0.1)B2       |20.1.1.x  
        |         |                                 |       |
        | Addr    A3(160.0.0.1) --------(170.0.0.1)B3 Addr  |11.2.2.x   
        | 1.1.1.1 |                              /  |2.2.2.2|
        +---------+                             /   +-------+ 
                   \                           /
                    \                         /PaymentFlow
                     \                       /
                      \                +----+----+ 
                       +---------------| payment |				
                                       | Gateway |
                                       +---------+
]]></artwork>
	</figure> 
</section>
<section title="4.2. Constrained Propagation of Edge Capability">
	<t>BGP has a built-in mechanism [RFC4684] to dynamically achieve the constrained distribution of edge information. In a nutshell, an SD-WAN edge sends RT Constraint (RTC) NLRI to the RR for the RR to install an outbound route filter, as shown in the figure below:</t>
	
	<figure
       anchor="Constraint propagation of Edge Property"
       title="Constraint propagation of Edge Property">
       <artwork><![CDATA[ 
    RT Constraint                   RT constraint 
    NLRI={SD-WAN#1, SD-WAN#2}         NLRI={SD-WAN#1, SD-WAN#3}
            ----->                 +---+      <-----------
              +--------------------|RR1|------------------+
              | Outbound Filter    +---+  Outbound Filter |
              | Permit SD-WAN#1,#2      Permit SD-WAN#1,#3|
              | Deny all                 Deny all         |
              |   <-------                --------->      |
              |                                           |
        +-----+---+  MPLS Path                      +-----+-+   
11.1.1.x| C-PE1   A1-------------------------------B1 C-PE2 |10.1.1.x
        |         |                                 |       |
21.1.1.x|         A2(192.10.0.10)------( 192.0.0.1)B2       |20.1.1.x  
        |         |                                 |       |
        | Addr    A3(160.0.0.1) --------(170.0.0.1)B3 Addr  |   
        | 1.1.1.1 |                                 |2.2.2.2|
        +---------+                                 +-------+  
SD-WAN VPN #1                                          SD-WAN VPN #1
SD-WAN VPN #2                                          SD-WAN VPN #3
]]></artwork>
	</figure> 
	<t>However, a SD-WAN overlay network can span across untrusted networks, RR cannot trust the RT Constraint (RTC) NLRI BGP UPDATE from any nodes. RR can only process the RTC NLRI from authorized peers for a SD-WAN VPN.</t> 
	<t>It is out of the scope of this document on how RR is configured with the policies to filter out unauthorized nodes for specific SD-WAN VPNs.</t> 
	<t>When the RR receives BGP UPDATE from an edge node, it propagates the received UPDATE message to the nodes that are in the Outbound Route filter for the specific SD-WAN VPN.</t>  
</section>
</section>
<section title="Client Route UPDATE">
<t>The SD-WAN network's Client Route UPDATE message is the same as the L3 VPN or EVPN client route UDPATE message. The SD-WAN Client Route UPDATE message uses the Encapsulation Extended Community and the Color Extended Community to link with the SD-WAN Underlay UPDATE Message.</t>
 <section title="SD-WAN VPN ID in Client Route Update">  
 <t>An SD-WAN VPN is same as a client VPN in a BGP controlled SD-WAN network. The Route Target Extended Community should be included in a Client Route UPDATE message to differentiate the client routes from routes belonging to other VPNs.</t>
 </section>
  <section title="SD-WAN VPN ID in Data Plane">  
	<t>For an SD-WAN edge node which can be reached by both MPLS and IPsec paths, the client packets reached by MPLS network will be encoded with the MPLS Labels based on the scheme specified by [RFC8277].</t> 
	<t>For GRE Encapsulation within an IPsec tunnel, the GRE key field can be used to carry the SD-WAN VPN ID. For network virtual overlay (VxLAN, GENEVE, etc.) encapsulation within the IPsec tunnel, the Virtual Network Identifier (VNI) field is used to carry the SD-WAN VPN ID.</t>
 </section>
 </section>
 
 <section title="SD-WAN Underlay UPDATE">
 <t>The hybrid underlay tunnel UPDATE is to advertise the detailed properties associated with the 
 public facing WAN ports and IPsec tunnels. The Edge node will advertise its own properties to 
 its designated RR via the secure connection.</t>
 <section title="NLRI for SD-WAN Underlay Tunnel Update">
	<t>A new NLRI SAFI[RFC5521](SD-WAN SAFI=74)is introduced within the MP_REACH_NLRI Path Attribute of RFC4760 for advertising the detailed properties of the SD-WAN tunnels terminated at the edge node:</t>
 	<figure
      anchor="SD-WAN NLRI"
      title="SD-WAN NLRI">
        <artwork><![CDATA[  
+------------------+
|    Route Type    | 2 octet
+------------------+  
|     Length       | 2 Octet
+------------------+   
|   Type Specific  |  
~ Value (Variable) ~  
|                  |  
+------------------+
]]></artwork>
</figure>
<t>Where</t>
<t>
	<list style="hanging">
	<t hangText="Route (NLRI) Type:"> 2 octet value to define the encoding of the rest of the SD-WAN the NLRI.
	</t>
	<t hangText="Length:">2 octets of length expressed in bits as defined in [RFC4760].</t>
	</list>
</t>
<t></t>
<t>This document defines the following SD-WAN Route type:</t>
     <t>
	<list style="hanging">
	 <t hangText="NLRI Route-Type= 1:"> For advertising the detailed properties of the SD-WAN 
	 tunnels terminated at the edge, where the transport network port can be uniquely identified by a 
	 tuple of three values (Port-Local-ID, SD-WAN-Color, SD-WAN-Node-ID). 
	 The SD-WAN NLRI Route-Type =1 has the following encoding:
	<figure
           anchor="SD-WAN NLRI Route Type 1"
           title="SD-WAN NLRI Route Type 1">
           <artwork><![CDATA[  
     +------------------+
     |  Route Type = 1  | 2 octet
     +------------------+
     |     Length       | 2 Octet
     +------------------+
     |   Port-Local-ID  | 4 octets
     +------------------+
     |   SD-WAN-Color   | 4 octets
     +------------------+
     |  SD-WAN-Node-ID  | 4 or 16 octets
     +------------------+
		]]></artwork>
	</figure>
	</t>
	<t hangText="Port local ID:"> SD-WAN edge node Port identifier, which is locally significant. 
		If the SD-WAN NLRI applies to multiple WAN ports, this field is NULL.</t>
	<t hangText="SD-WAN-Color:">to represent a group of tunnels, which correlate 
		with the Color-Extended-community included in the client routes UPDATE. 
		When a client route can be reached by multiple SD-WAN edges co-located at one site, 
		the SD-WAN- Color can represent a group of tunnels terminated at those SD-WAN edges co-located at the site, 
		which effectively represent the site.</t>
	<t hangText="SD-WAN Node ID:"> The node's IPv4 or IPv6 address.</t>
	</list>
	</t>
	<t> Route Type values outside of 1 are out of scope for this document. 
	 The Route Type allows for other existing SD-WAN applications to use this basic format. 	 
	</t>
 </section>
 
 <section title="SD-WAN-Hybrid Tunnel Encoding">
<t> A new BGP Tunnel-Type=SD-WAN-Hybrid (code point 25) is to indicate hybrid underlay tunnels.</t>
 	<figure
           anchor="SD-WAN Hybrid Value Field"
           title="SD-WAN Hybrid Value Field">
           <artwork><![CDATA[  
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Tunnel-Type=25(SD-WAN-Hybrid )| Length (2 Octets)             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           Sub-TLVs                            |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

		]]></artwork>
	</figure>
 </section>
 
  <section title="IPsec-SA-ID Sub-TLV">
  <t>IPsec-SA-ID Sub-TLV within the Hybrid Underlay Tunnel UPDATE indicates one or more pre-established IPsec SAs by using their identifiers, instead of listing all the detailed attributes of the IPsec SAs.</t> 
  <t>Using an IPsec-SA-ID Sub-TLV not only greatly reduces the size of BGP UPDATE messages, but also allows the pairwise IPsec rekeying process to be performed independently.</t>
  <t>The following is the structure of the IPsec-SA-ID sub-TLV:</t>
  <figure
    anchor="IPsec-SA-ID Sub-TLV"
    title="IPsec-SA-ID Sub-TLV">
     <artwork><![CDATA[  
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Type=64 (IPsec-SA-ID subTLV)   |  Length (2 Octets)            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
|                      IPsec SA Identifier #1                   |      
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
|                      IPsec SA Identifier #2                   |      
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
		]]></artwork>
	</figure>
  </section> 
  
  <section title="Extended Port Attribute Sub-TLV">
  <t>Extended Port Attribute Sub-TLV is to advertise the properties associated with a public Internet-facing WAN port that might be behind NAT. An SD-WAN edge node can query a STUN Server (Session Traversal of UDP through Network address translation [RFC8489]) to get the NAT properties, including the public IP address and the Public Port number, to pass to its peers.</t>
  <t>The location of a NAT device can be:
  	<list style="symbol">
		<t>Only the initiator is behind a NAT device. Multiple initiators can be behind separate NAT devices. Initiators can also connect to the responder through multiple NAT devices.</t>
		<t>Only the responder is behind a NAT device.</t>
		<t>Both the initiator and the responder are behind a NAT device.</t>
	</list>
  </t>
 <t>The initiator's address and/or responder's address can be dynamically assigned 
  by an ISP or when their connection crosses a dynamic NAT device that allocates 
  addresses from a dynamic address pool.</t> 
 <t>As one SD-WAN edge can connect to multiple peers, the pair-wise NAT exchange as IPsec's 
 IKE[RFC7296] is not efficient. In the BGP Controlled SD-WAN, NAT properties for a WAN port are 
 encoded in the Extended Port Attribute sub-TLV, which the following format:</t>
  <figure
    anchor="Extended Port Attribute  Sub-TLV"
    title="Extended Port Attribute  Sub-TLV">
     <artwork><![CDATA[  
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |Type=65(extPort| ExtPort Length| Reserved      |I|O|R|R|R|R|R|R|
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | NAT Type      |  Encap-Type   |Trans networkID|     RD ID     |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                  Local  IP Address                            | 
            32-bits for IPv4, 128-bits for Ipv6
                    ~~~~~~~~~~~~
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                  Local  Port                                  |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                Public IP                                      | 
            32-bits for IPv4, 128-bits for Ipv6
                    ~~~~~~~~~~~~
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                Public Port                                    |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
    |               Extended SubSub-TLV                             | 
    ~                                                               ~
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
		]]></artwork>
	</figure>
	<t>Where:
	<list style="symbol">
		<t>Extended Port Attribute Type (=65): indicating it is the Extended Port Attribute SubTLV</t>
		<t>ExtPort Length: the length of the subTLV. </t>
		<t>Flags:
			<list style="symbol">
			<t>I bit (CPE port address or Inner address scheme):
			    <list> 
				<t>If set to 0, indicate the inner (private) address is IPv4.</t> 
				<t>If set to 1, indicates the inner address is IPv6. </t>
				</list>
			</t>
			<t>O bit (Outer address scheme):
			    <list>
				<t>If set to 0, indicate the inner (private) address is IPv4.</t> 
				<t>If set to 1, indicates the inner address is IPv6. </t>
				</list>
			</t>
			<t>R bits: reserved for future use. Must be set to 0 now:</t>
			</list>
		</t>
		<t>NAT Type: the NAT type can be one of the following values: 
		<list style="symbols"> 
		<t>1: without NAT ;  </t>
		<t>2: 1-to-1 static NAT; </t>
		<t>3: Full Cone; </t> 
		<t>4: Restricted Cone; </t>
		<t>5: Port Restricted Cone; </t>
 		<t>6: Symmetric; or </t> 
		<t>7: Unknown (i.e. no response from the STUN server).</t>
		</list>
		</t>
		<t>Encap-Type: the supported encapsulation types for the port. 
		<list>
		<t>Encap-Type=1: GRE;</t>
		<t>Encap-Type=2: VxLAN;</t> 
		</list> 
		Notes:
		<list>
		<t>The Encap-Type inside the Extended Port Attribute Sub-TLV is different from the RFC9012's 
		BGP-Tunnel-Encapsulation type (https://www.iana.org/assignments/bgp-tunnel-encapsulation/bgp-tunnel-encapsulation.xhtml#tunnel-types). The Extended Port Attribute Sub-TLV is a subTLV attached to the Tunnel Type TLV (the BGP-Tunnel-Type = 25 for the SD-WAN Hybrid tunnels). The port can indicate the specific encapsulations, such as:</t>
		<t>If the IPsec-SA-ID subTLV or the IPsec SA detailed subTLVs(Nonce/publicKey/Proposal) 
		are included in the SD-WAN-Hybrid tunnel, the Encap-Type indicates the 
		encapsulation type within the IPsec payload.</t>
		<t>If the IPsec SA subTLVs are not included in the SD-WAN-Hybrid Tunnel, 
		the Encap-Type indicates the encapsulation of the payload without IPsec encryption.</t> 
		</list> 
		</t>
		<t>Transport Network ID: Central Controller assigns a global unique ID to each transport network.</t>
		<t>RD ID: Routing Domain ID, need to be globally unique.
		<list>
		<t>Some SD-WAN deployment might have multiple levels, zones, or regions that are represented as routing domains. 
		Policies can govern if tunnels can be established across domains.  
		E.g., a hub node can establish tunnels with different domains; 
		but the spoke nodes cannot establish tunnels with nodes in different domains.</t>
		</list>
		</t>
		<t>Local IP: The local (or private) IP address of the WAN port.</t>
		<t>Local Port: used by Remote SD-WAN edge node for establishing IPsec to this specific port.</t>
		<t>Public IP: The IP address after the NAT. If NAT is not used, this field is set to NULL.</t>
		<t>Public Port: The Port after the NAT. If NAT is not used, this field is set to NULL.</t>
		<t>Extended SubSub-TLV: for carrying additional information about the underlay networks.</t> 
	</list>
	</t>
  </section> 
  <section title="Extended SubSub-TLV">
  <t>Two types of the Extended SubSub-TLVs are specified in this document: Underlay Network Transport SubSub-TLV and the underlay Geo Location SubSub-TLV.</t>
    <section title="Underlay Network Transport SubSub-TLV">
	<t>The Underlay Network Transport SubSub-TLV is an optional Sub-TLV to 
	carry the WAN port connection types and bandwidth, such as LTE, DSL, Ethernet, etc.</t>
	<t>The format of this Sub-TLV is as follows:
	<figure
    anchor="Underlay Network SubSub-TLV"
    title="Underlay Network SubSub-TLV">
     <artwork><![CDATA[  
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | UnderlayType  |      Length   |      Flag     |    Reserved   | 
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |Connection Type|   Port Type   |        Port Speed             |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
		]]></artwork>
	</figure>
	</t>
<t>Where:
<list style="hanging">
	  <t hangText="Underlay Network Properties:"> sub Type=66 </t>
      <t hangText="Length:"> always 6 bytes</t>
      <t hangText="Flag:"> a 1 octet value.</t>
      <t hangText="Reserved:"> 1 octet of reserved bits. It SHOULD be set to zero on
      transmission and MUST be ignored on receipt.</t>
	  <t hangText="Connection Type:"> are listed below as:
			<list style="symbol">
			<t>1 = Wired</t>
			<t>2 = WIFI</t>
			<t>3 = LTE </t>
			<t>4 = 5G </t>
		    </list>
	  </t>
	  <t hangText="Port Type:">Port type define as follows: 
			<list style="symbol">
			<t>1 = Ethernet </t>
			<t>2 = Fiber Cable </t>
			<t>3 = Coax Cable </t>
			<t>4 = Cellular </t>
			</list>
	  </t>
	 <t hangText="Port Speed:">The port seed is defined as 2 octet value. The values are defined as Gigabit speed.
	  For example, a value of 1 would mean 1 gigabit.</t>
</list>
</t>
</section> 
 </section> 
 </section>
 <section title="IPsec SA Property Sub-TLVs"> 
<t> This section describes the detailed IPsec SA properties sub-TLVs. 
When the IPsec SA properties are associated with the node, any of the node's WAN ports can be the 
outer destination address of the IPsec encapsulated data packets.</t>
 <section title="IPsec SA Nonce Sub-TLV"> 
 <t>The Nonce Sub-TLV is based on the Base DIM sub-TLV as described the Section 10.1 of [SECURE-EVPN].  The following fields are removed because:</t> 
	<list>
	<t>-  the Originator ID is same as the Node-ID in the SD-WAN NLRI,</t> 
	<t>-  the Tenant ID and Subnet ID are represented by the SD-WAN VPN ID in the Client UPDATE.</t>
	</list>
    <t>The format of this Sub-TLV is as follows:</t>
	<figure
    anchor="IPsec SA Nouce Sub-TLV"
    title="IPsec SA Nouce Sub-TLV">
     <artwork><![CDATA[  
        0                   1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |   ID Length   |       Nonce Length            |I|   Flags     |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                             Rekey                             |
       |                            Counter                            |
       +---------------------------------------------------------------+
       |                IPsec SA Identifier                            |
       +---------------------------------------------------------------+
       |                                                               |
       ~                          Nonce Data                           ~
       |                                                               |
       +---------------------------------------------------------------+
		]]></artwork>
	</figure>
	<t> where:
	<list> 
	<t>IPsec SA ID - The 4 bytes IPSec SA ID is to differentiate multiple IPsec SAs terminated at the edge.
    <list>
	<t>The IPsec SA ID can be used in the IPsec-SA-ID subTLV of a different BGP UPDATE message to refer to 
	all the values carried in the IPsec Public Key SubTLV and the IPsec SA Proposal Sub-TLV that are in the 
	same BGP UPDATE message as the IPsec SA Nonce sub-TLV.</t>  
	</list>
	</t>
	</list>
	</t>
 </section>
 <section title="IPsec Public Key Sub-TLV"> 
 <t>The IPsec Public Key Sub-TLV is derived from the Key Exchange Sub-TLV described in [SECURE-EVPN] with an addition of Duration filed to define the IPSec SA life span. The edge nodes would pick the shortest duration value advertised by the peers.</t>
<t>The format of this Sub-TLV is as follows:</t>
	<figure
    anchor="IPsec SA Public Key Sub-TLV"
    title="IPsec SA Public Key Sub-TLV">
     <artwork><![CDATA[  
        0                   1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |   Diffie-Hellman Group Num    |          Reserved             |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                                                               |
       ~                       Key Exchange Data                       ~
       |                                                               |
       +---------------------------------------------------------------+
       |                            Duration                           |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
		]]></artwork>
	</figure>
 </section>
  <section title="IPsec SA Proposal Sub-TLV"> 
  <t>The IPsec SA Proposal Sub-TLV is to indicate the number of Transform Sub-TLVs. This Sub-TLV aligns with the sub-TLV structure from [SECURE-EVPN].</t>
  	<figure
    anchor="IPsec SA Proposal Sub-TLV"
    title="IPsec SA Proposal Sub-TLV">
     <artwork><![CDATA[  
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|    Transform Attr Length      |Transform Type |    Reserved.  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|     Transform ID              |          Reserved             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
~                        Transform Attributes                   ~
|                                                               |
+---------------------------------------------------------------+
		]]></artwork>
	</figure>
	<t>The Transform Type and the Transform Attributes Sub-sub-TLV are taken from the section 3.3.2 and 3.3.5 of RFC7296, respectively.</t>
  </section>
  <section title="Simplified IPsec SA sub-TLV"> 
  <t>For a simple SD-WAN network with edge nodes supporting only a few pre-defined encryption algorithms, a simple IPsec sub-TLV can be used to encode the pre-defined algorithms, as below:</t>
  	<figure
    anchor="Simplifed IPsec SA Sub-TLV"
    title="Siplified IPsec SA Sub-TLV">
     <artwork><![CDATA[  
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |IPsec-simType  |IPsecSA Length                 | Flag          |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | Transform     | Mode          | AH algorithms |ESP algorithms |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |         ReKey Counter (SPI)                                   |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | key1 length   |         Key 1                                 ~
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
    | key2 length   |         Key 2                                 ~
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | key-i length  |         Nonce                                 ~
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |        Duration                                               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
		]]></artwork>
	</figure>
  <t>Where:</t>
  <list style="hanging">
  <t hangText="IPsec-SimType=70:"> indicate the simplified IPsec SA attributes.</t> 
  <t hangText="IPsec-SA subTLV Length (2 Byte):"> 25 (or more).</t>
  <t hangText="Flags:"> 1 octet of flags. None are defined at this stage. 
  Flags SHOULD be set to zero on transmission and MUST be ignored on receipt.</t>
  <t hangText="Transform (1 Byte):"> 
	<list style="symbol">
		<t>Transform = 1 means AH,</t> 
		<t>Transform = 2 means ESP, or</t> 
		<t>Transform = 3 means AH+ESP.</t> 
	</list>
  </t>
	<t hangText="IPsec Mode (1 byte):">
	<list style="symbol">
		<t>Mode = 1 indicates that the Tunnel Mode is used.</t>
		<t>Mode = 2 indicates that the Transport mode is used.</t>
	</list>
	</t>
	<t hangText="AH algorithms (1 byte):"> AH authentication algorithms supported. The values are specified by [IANA-AH]. Each SD-WAN edge node can support multiple authentication algorithms; send to its peers to negotiate the strongest one.</t> 	
	<t hangText="ESP algorithms (1 byte):"> the ESP algorithms supported. Its values are specified by [IANA-ESP]. One SD-WAN edge node can support multiple ESP algorithms and send them to its peers to negotiate the strongest one. The default algorithm is AES-256.
	<list>
	<t>	When a node supports multiple authentication algorithms, the initial UPDATE needs to include the "Transform Sub-TLV" described by [SECURE-EVPN] to describe all of the algorithms supported by the node.</t> 
	</list>
	</t>
	<t hangText="Rekey Counter (Security Parameter Index)):"> 4 bytes</t>
	<t hangText="Public Key:"> IPsec public key</t>
	<t hangText="Nonce:"> IPsec Nonce</t>
	<t hangText="Duration:"> SA life span.</t> 
  </list>
  </section>
 </section>
 <section title="Error and Mismatch Handling"> 
 <section title="Color Mismatch"> 
 <t>When an SD-WAN edge receives a client route BGP UPDATE from a peer with a color that does not match with any of the tunnels advertised by the peer, the client route UPDATE should be ignored, and an error message (e.g., Syslog) should be generated to its management system per[RFC7606].</t> 
 <t>For example, for two peers, PA and PB, both PA and PB will first advertise their SD-WAN properties (i.e., tunnel properties). Say PA advertises two SD-WAN tunnels (Red and Green), and PB advertises two SD-WAN tunnels (Yellow and Purple). PB should report a mismatch error message when PB receives a Client Update from PA with a color NOT Red or Green. PA should report a Mismatch Error when PA receives a Client Update from PB with a color that is not Yellow and Purple.</t>
 <t>Upon receiving a Tunnel Update that includes the IPsec-SA-ID subTLV from a peer, the BGP node should report Mismatch error if the IPsec SA has not been established yet.</t>
 <t>Moreover, if the encap-Types, in the Extended Port Attributes Sub-TLV, in the received SDWAN update is not supported by the local ports, the corresponding ports between the remote edge and local edge will not establish an overlay tunnel. Overlay tunnels would only be established between two ports belonging to different edges, if their attributes are compatible. For instance, the encap Types should match. Policies and configurations outside the scope of this document could allow for mismatched attributes to be present and allow establishing overlay tunnels.</t>
 </section>
 <section title="IPsec Attributes Mismatch"> 
 <t>Each C-PE device advertises a SD-WAN SAFI Underlay NLRI to the other C-PE devices via a 
 BGP Route Reflector to establish pairwise SAs between itself and every other remote C-PEs. 
 During the SAFI NLRI advertisement, the BGP originator would include either simple IPSec 
 Security Association properties defined in IPSec SA Sub-TLV based on IPSec-SA-Type = 1 or 
 full-set of IPSec Sub-TLVs including Nonce, Public Key, Proposal and number of Transform Sub-TLVs 
 based on IPSec-SA-Type = 2.</t>
 <t>The C-PE devices compare the IPSec SA attributes between the local and remote WAN ports. 
 If there is a match on the SA Attributes between the two ports, the IPSec Tunnel is established.</t>
 <t>The C-PE devices would not try to negotiate the base IPSec-SA parameters between the local 
 and the remote ports in the case of simple IPSec SA exchange or the Transform sets between local 
 and remote ports if there is a mismatch on the Transform sets in the case of full-set of IPSec SA Sub-TLVs.</t>
<t>Here is an example of using Figure 1 in Section 3 to establish an IPsec Tunnel between 
C-PE1 and C-PE2 WAN Ports A2 and B2 (A2: 192.10.0.10 - B2:192.0.0.1).</t>
 <t>C-PE1 needs to advertise the following attributes for establishing the IPsec SA:</t>
<t> 
 <list style="symbols">
 <t>NextHop: 192.10.0.10</t>
 <t>SD-WAN Node ID</t>
 <t>SD-WAN-Site-ID</t> 
 <t>Tunnel Encap Attr (Type=SD-WAN) - 
	<list style="symbols">
	<t>Extended Port Attribute Sub-TLV 
	<list>
	<t>Transport SubSubTLV - with information on ISP3. 
	</t>
    </list>
    </t>	
	<t>Transport-Sub-TLV for detailed information about the ISP3</t>
	<t>IPsec SA Nonce Sub-TLV,</t> 
	<t>IPsec SA Public Key Sub-TLV,</t>
	<t>Proposal Sub-TLV with Num Transforms = 1</t>
	<t>{Transforms Sub-TLV - Trans 1}</t>
	</list>	
 </t>
 </list>
 </t>
<t>C-PE2 needs to advertise the following attributes for establishing IPsec SA:
</t>
<t>
<list style="hanging">
<t hangText="NH:"> 192.0.0.1</t>
<t hangText="SD-WAN Node ID:"> (value) </t>
<t hangText="SD-WAN-Site-ID:"> (value) </t>
<t hangText="Tunnel Encap Attr (Type=SD-WAN)">
	<list style="symbols">
	<t>Extended Port Attribute Sub-TLV 
	<list>
	<t>Transport SubSubTLV - with information on ISP1. 
	</t>
    </list> 
    </t> 	
	<t>IPsec SA Nonce Sub-TLV, </t>
	<t>IPsec SA Public Key Sub-TLV,</t>
	<t>Proposal Sub-TLV with Num Transforms = 1</t>
	<t>{Transforms Sub-TLV - Trans 2}</t>
	</list>	
</t>
</list>
</t>
<t>As there is no matching transform between the WAN ports A2 and B2 in C-PE1 and C-PE2, 
respectively, no IPsec Tunnel will be established.</t>
 </section>
 </section>
 <section title="SD-WAN BGP UPDATE Encoding Examples"> 
 <section title="Encoding example of WAN Port properties"> 
 <t>The C-PE2 of Figure 1 can send the following SD-WAN UPDATE messages to 
 advertise the properties associated with WAN Port 192.0.0.1 and WAN Port 170.0.0.1, 
 respectively:</t>
 <t>
 <list style="hanging">
 <t hangText="SD-WAN NLRI:"> AFI=IPv4/IPv6 and SAFI = SD-WAN 
	<list style="symbols">
	<t>Color match with the Client route UPDATE's Color Extended Community[RFC4360]</t>
	<t>local port id for WAN port 192.0.0.1</t>
	<t>Node-ID= 2.2.2.2 (C-PE2)</t>
	</list>
 </t>
 <t hangText="Tunnel-Type:"> Hybrid-SD-WAN</t>
 <t hangText="Extended-Port-SubTLV:"> for 192.0.0.1</t>
 <t hangText="SD-WAN NLRI:"> AFI=IPv4/IPv6 and SAFI = SD-WAN 
 	<list style="symbols">
	<t>Color match with the Client route UPDATE's Color Extended Community</t>
	<t>local port id for WAN port 170.0.0.1</t>
	<t>Node-ID= 2.2.2.2 (C-PE2)</t>
	</list>
 </t>
 <t hangText="Tunnel-Type:">Hybrid-SD-WAN</t>
 <t hangText="Extended-Port-SubTLV"> for 170.0.0.1</t>
 </list>
 </t>
 </section>
 <section title="Encoding example of IPsec SA terminated at the C-PE2"> 
 <t>The C-PE2 of the Figure 1 can send the following SD-WAN UPDATE messages 
 to advertise node level IPsec SA:</t> 
 <t>
 <list style="hanging">
 <t hangText="SD-WAN NLRI:"> AFI=IPv4/IPv6 and SAFI = SD-WAN
	<list style="symbols"> 	
	<t>Color match with the Client route UPDATE's Color Extended Community</t>
	<t>Port-ID=0</t>
	<t>Node-ID= 2.2.2.2 (C-PE2)</t>
	</list>
</t>
<t hangText="Tunnel-Type:"> Hybrid-SD-WAN</t>
<t hangText="IPsec-SA-ID Sub-TLV or IPsec SA Property Sub-TLVs"> </t>
</list>
</t>
 </section>
 <section title="Encoding example of using IPsec-SA-ID Sub-TLV"> 
 <t>Here is an encoding example of four IPsec SAs terminated at the same node.</t>
<t>
	<figure
    anchor="Encoding Example of 4 IPsec SA"
    title="Encoding Example of 4 IPsec SA">
     <artwork><![CDATA[  
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
| Tunnel-Type =SD-WAN-Hybrid    |       Length =                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
|      Tunnel-end-point sub-TLV                                 | 
~                                                               ~ 
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
| subTLV-Type = IPsec-SA-ID     |       Length =                | 
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                        
|                     IPsec SA Identifier = 1                   | 
+---------------------------------------------------------------+  
|                     IPsec SA Identifier = 2                   | 
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+   
|                     IPsec SA Identifier = 3                   | 
+-------------------------------+-------------------------------+    
|                     IPsec SA Identifier = 4                   |
+---------------------------------------------------------------+  

		]]></artwork>
	</figure>
</t>
 <t>The Length of the Tunnel-Type = SDDWAN-Hybrid is the sum of the following:
 <list style="symbols">
 <t>Tunnel-end-point sub-TLV total length: </t>
 <t>The IPsec-SA-ID Sub-TLV length </t> 
</list>
</t>
 </section>
 </section>
 <section title="Manageability Considerations"> 
 <t>Unlike MPLS VPN whose PE nodes are all controlled by the network operators, 
 SD-WAN edge nodes can be installed anywhere, in shopping malls, in 3rd party Cloud DCs, etc.</t> 
 <t>It is very important to ensure that client routes advertisement from an SD-WAN edge 
 node are legitimate. The RR needs to drop all the BGP Update messages from an SD-WAN 
 edge nodes that have invalid Route Targets.</t>   

 </section>
  <section title="Security Considerations"> 
 <t>The document describes the encoding for SD-WAN edge nodes to advertise its properties 
 to their peers to its RR, which propagates to the intended peers via untrusted networks.</t> 
 <t>The secure propagation is achieved by secure channels, such as TLS, SSL, or IPsec, 
 between the SD-WAN edge nodes and the local controller RR.</t> 
 <t>SD-WAN edge nodes might not have secure channels with the RR. In this case, 
 BGP connection has be established over IPsec or TLS.</t>  
 </section>
 
  <section title="IANA Considerations"> 
   <section title="Hybrid (SD-WAN) Overlay SAFI"> 
	<t>IANA has assigned SAFI = 74 as the Hybrid (SD-WAN) SAFI.</t>
   </section>
    <section title="Tunnel Encapsulation Attribute Type"> 
	<t>IANA is requested to assign a type from the BGP Tunnel Encapsulation Attribute Tunnel Types as follows [RFC8126]:</t>
	     <artwork><![CDATA[  
 Value   Description    Reference
-----   ------------   ---------
 25	  SD-WAN-Hybrid   (this document)
		]]></artwork>
	</section>
	  <section title="Tunnel Encapsulation Attribute Sub-TLV Types"> 
	  <t>IANA is requested to assign the following sub-Types in the BGP Tunnel Encapsulation Attribute Sub-TLVs registry:</t>
	  	     <artwork><![CDATA[  
   Value   Type Description               Reference
   -----   -----------------------   ---------------
    64	IPSEC-SA-ID Sub-TLV             (Section 6.3)
    65	Extended Port Property Sub-TLV  (Section 6.4)
    66	Underlay Transport Sub-TLV      (Section 6.5)
    67	IPsec SA Nonce Sub-TLV          (Section 7.1)
    68	IPsec Public Key Sub-TLV        (Section 7.2)
    69	IPsec SA Proposal Sub-TLV       (Section 7.3)
    70	Simplified IPsec SA sub-TLV     (Section 7.4)
		]]></artwork>
	  </section>
 </section>
 
</middle>	

  <back>
    <references title="Normative References">
     
      <?rfc include="reference.RFC.2119"?>
	  <?rfc include="reference.RFC.4271"?>
	  <?rfc include="reference.RFC.4360"?>
	  <?rfc include="reference.RFC.4364"?>
	  <?rfc include="reference.RFC.4760"?>
	  <?rfc include="reference.RFC.7296"?>
	  <?rfc include="reference.RFC.7606"?>
	  <?rfc include="reference.RFC.8126"?>
      <?rfc include="reference.RFC.8174"?>
      <?rfc include="reference.RFC.8277"?>
	  <?rfc include="reference.RFC.8489"?>
      <?rfc include="reference.RFC.9012"?>

    </references>

    <references title="Informative References">
	  
	  <?rfc include="reference.RFC.4684"?>
	  <?rfc include="reference.RFC.5521"?>
	  <?rfc include="reference.RFC.9016"?>
 

	 
	 <reference anchor="SD-WAN-BGP-USAGE" target="https://datatracker.ietf.org/doc/draft-ietf-bess-bgp-sdwan-usage/">
	 <front>
	    <title> BGP Usage for SD-WAN Overlay Networks</title>
		<author>
		 <organization>L. Dunbar, A Sajassi, J Drake, and B. Najem</organization>
		</author>
		<date month="Sept" year="2023"/>
     </front>
	 </reference>
	 <reference anchor="Net2Cloud" target= "https://datatracker.ietf.org/doc/draft-ietf-rtgwg-net2cloud-problem-statement/">
	 <front>
	    <title> Dynamic Networks to Hybrid Cloud DCs: Problem Statement and Mitigation Practice</title>
		<author>
		 <organization>L. Dunbar, A Malis, C. Jacquenet, M. Toy and  K. Majumdar</organization>
		</author>
		<date month="Sept" year="2023"/>
     </front>
	 </reference>
	
	<reference anchor="IANA-AH" target="https://www.iana.org/assignments/isakmp-registry/isakmp-registry.xhtml#isakmp-registry-9">
	 <front>
	    <title> IANA-AH </title>
		<author>
		 <organization>IANA</organization>
		</author>
     </front>
	 </reference>
	 
	 <reference anchor="IANA-ESP" target="https://www.iana.org/assignments/isakmp-registry/isakmp-registry.xhtml#isakmp-registry-9">
	 <front>
	    <title> IANA-ESP </title>
		<author>
		 <organization>IANA</organization>
		</author>
     </front>
	 </reference>
	 
	 
	<reference anchor="SECURE-EVPN" target="https://datatracker.ietf.org/doc/draft-ietf-bess-secure-evpn/">
	 <front>
	    <title> Secure EVPN </title>
		<author>
		 <organization>A Sajassi, et al</organization>
		</author>
		<date month="July" year="2023"/>
     </front>
	 </reference>
	 
	 
    </references>
	<section title="Acknowledgments">
	<t>Acknowledgements to Wang Haibo, Shunwan Zhuang, Hao Weiguo, and ShengCheng for implementation contribution; 
	Many thanks to Yoav Nir, Graham Bartlett, Jim Guichard, John Scudder, and Donald Eastlake for their 
	review and suggestions.</t>

	</section>
	<section numbered="false" toc="include" removeInRFC="false" pn="section-appendix.b">
      <name slugifiedName="name-contributors">Contributors</name>
   <t>Below is a list of other contributing authors:</t>
   <t>
      	<contact fullname="Gyan Mishra">
        <organization showOnFrontPage="true">Verizon</organization>
        <address>
          <postal>     
            <country>USA</country>
          </postal>
          <email>gyan.s.mishra@verizon.com</email>
        </address>
      </contact>
	  
	  <contact fullname="Shunwan Zhuang">
        <organization showOnFrontPage="true">Huawei</organization>
        <address>
          <postal>
            <city>Beijing</city>
            <country>China</country>
          </postal>
          <email>zhuangshunwan@huawei.com</email>
        </address>
      </contact>
	  
      <contact fullname="Sheng Cheng">
        <organization showOnFrontPage="true">Huawei</organization>
        <address>
          <postal>     
            <city>Beijing</city>
            <country>China</country>
          </postal>
          <email>shengcheng@huawei.com</email>
        </address>
      </contact>

	  
	  <contact fullname="Donald Eastlake">
        <organization showOnFrontPage="true">Futurewei</organization>
        <address>
          <postal>     
            <country>USA</country>
          </postal>
          <email>d3e3e3@gmail.com</email>
        </address>
      </contact>
   </t> 
    </section>
  </back>
</rfc>
