Applicability of EVPN to NVO3 Networks

AC: Attachment Circuit or logical interface associated to a given BT. To determine the AC on which a packet arrived, the NVE will examine the physical/logical port and/or VLAN tags (where the VLAN tags can be individual c-tags, s-tags or ranges of both). ARP and ND: Address Resolution Protocol and Neighbor Discovery protocol. BD: or Broadcast Domain, it corresponds to a tenant IP subnet. If no suppression techniques are used, a BUM frame that is injected in a BD will reach all the NVEs that are attached to that BD. An EVI may contain one or multiple BDs depending on the service model . This document will use the term BD to refer to a tenant subnet. BT: a Bridge Table, as defined in . A BT is the instantiation of a BD in an NVE. When there is a single BD on a given EVI, the MAC-VRF is equivalent to the BT on that NVE. BUM: Broadcast, Unknown unicast and Multicast frames. CLOS: a multistage network topology described in , where all the edge switches (or Leafs) are connected to all the core switches (or Spines). Typically used in Data Centers nowadays. DF and NDF: they refer to Designated Forwarder and Non-Designated Forwarder, which are the roles that a given PE can have in a given ES. ECMP: Equal Cost Multi-Path. EVPN: Ethernet Virtual Private Networks, as described in . EVPN VLAN-based service model: one of the three service models defined in . It is characterized as a BD that uses a single VLAN per physical access port to attach tenant traffic to the BD. In this service model, there is only one BD per EVI. EVPN VLAN-bundle service model: similar to VLAN-based but uses a bundle of VLANs per physical port to attach tenant traffic to the BD. As in VLAN-based, in this model there is a single BD per EVI. EVPN VLAN-aware bundle service model: similar to the VLAN-bundle model but each individual VLAN value is mapped to a different BD. In this model there are multiple BDs per EVI for a given tenant. Each BD is identified by an "Ethernet Tag", that is a control-plane value that identifies the routes for the BD within the EVI. ES: Ethernet Segment. When a Tenant System (TS) is connected to one or more NVEs via a set of Ethernet links, then that set of links is referred to as an 'Ethernet segment'. Each ES is represented by a unique Ethernet Segment Identifier (ESI) in the NVO3 network and the ESI is used in EVPN routes that are specific to that ES. Ethernet Tag: Used to represent a BD that is configured on a given ES for the purpose of DF election. Note that any of the following may be used to represent a BD: VIDs (including Q-in-Q tags), configured IDs, VNIs (Virtual Extensible Local Area Network (VXLAN) Network Identifiers), normalized VIDs, I-SIDs (Service Instance Identifiers), etc., as long as the representation of the BDs is configured consistently across the multihomed PEs attached to that ES. The Ethernet Tag value MUST be different from zero. EVI: or EVPN Instance. It is a Layer-2 Virtual Network that uses an EVPN control-plane to exchange reachability information among the member NVEs. It corresponds to a set of MAC-VRFs of the same tenant. See MAC-VRF in this section. GENEVE: Generic Network Virtualization Encapsulation, an NVO3 encapsulation defined in . IP-VRF: an IP Virtual Routing and Forwarding table, as defined in . It stores IP Prefixes that are part of the tenant's IP space, and are distributed among NVEs of the same tenant by EVPN. Route-Distinghisher (RD) and Route-Target(s) (RTs) are required properties of an IP-VRF. An IP-VRF is instantiated in an NVE for a given tenant, if the NVE is attached to multiple subnets of the tenant and local inter-subnet-forwarding is required across those subnets. IRB: Integrated Routing and Bridging interface. It refers to the logical interface that connects a BD instance (or a BT) to an IP- VRF and allows to forward packets with destination in a different subnet. MAC-VRF: a MAC Virtual Routing and Forwarding table, as defined in . The instantiation of an EVI (EVPN Instance) in an NVE. Route-distinghisher (RD) and Route-Target(s) (RTs) are required properties of a MAC-VRF and they are normally different than the ones defined in the associated IP-VRF (if the MAC-VRF has an IRB interface). NVE: Network Virtualization Edge is a network entity that sits at the edge of an underlay network and implements L2 and/or L3 network virtualization functions. The network-facing side of the NVE uses the underlying L3 network to tunnel tenant frames to and from other NVEs. The tenant-facing side of the NVE sends and receives Ethernet frames to and from individual Tenant Systems. In this document, an NVE could be implemented as a virtual switch within a hypervisor, a switch or a router, and runs EVPN in the control-plane. NVO3 or Overlay tunnels: Network Virtualization Over Layer-3 tunnels. In this document, NVO3 tunnels or simply Overlay tunnels will be used interchangeably. Both terms refer to a way to encapsulate tenant frames or packets into IP packets whose IP Source Addresses (SA) or Destination Addresses (DA) belong to the underlay IP address space, and identify NVEs connected to the same underlay network. Examples of NVO3 tunnel encapsulations are VXLAN , GENEVE or MPLSoUDP . PE: Provider Edge router. PTA: Provider Multicast Service Interface Tunnel Attribute. RT and RD: Route Target and Route Distinguisher. RT-1, RT-2, RT-3, etc.: they refer to Route Type followed by the type number as defined in the IANA registry for EVPN route types. SA and DA: Source Address and Destination Address. They are used along with MAC or IP, e.g. IP SA or MAC DA. SBD: Supplementary Broadcast Domain. Defined in , it is a BD that does not have any ACs, only IRB interfaces, and provides connectivity among all the IP-VRFs of a tenant in the Interface-ful IP-VRF-to-IP-VRF models. TS: Tenant System. VNI: Virtual Network Identifier. Irrespective of the NVO3 encapsulation, the tunnel header always includes a VNI that is added at the ingress NVE (based on the mapping table lookup) and identifies the BT at the egress NVE. This VNI is called VNI in VXLAN or GENEVE, VSID in nvGRE or Label in MPLSoGRE or MPLSoUDP. This document will refer to VNI as a generic Virtual Network Identifier for any NVO3 encapsulation. VXLAN: Virtual eXtensible Local Area Network, an NVO3 encapsulation defined in .

This section discusses the applicability of EVPN to NVO3 networks. The intend is not to provide a comprehensive explanation of the protocol itself but give an introduction and point at the corresponding reference document, so that the reader can easily find more details if needed.

EVPN supports multiple Route Types and each type has a different function. For convenience, shows a summary of all the existing EVPN route types and its usage. We will refer to these route types as RT-x routes throughout the rest of the document, where x is the type number included in the first column of . Type Description Usage 1 Ethernet Auto-Discovery Multi-homing: Per-ES: Mass withdrawal, Per-EVI: aliasing/backup 2 MAC/IP Advertisement Host MAC/IP dissemination, supports MAC mobility and protection 3 Inclusive Multicast Ethernet Tag NVE discovery and BUM flooding tree setup 4 Ethernet Segment Multi-homing: ES auto-discovery and DF Election 5 IP Prefix IP Prefix dissemination 6 Selective Multicast Ethernet Tag Indicate interest for a multicast S,G or *,G 7 Multicast Join Synch Multi-homing: S,G or *,G state synch 8 Multicast Leave Synch Multi-homing: S,G or *,G leave synch 9 Per-Region I-PMSI A-D BUM tree creation across regions 10 S-PMSI A-D Multicast tree for S,G or *,G states 11 Leaf A-D Used for responses to explicit tracking

Although the applicability of EVPN to NVO3 networks spans multiple documents, EVPN's baseline specification is . allows multipoint layer-2 VPNs to be operated as IP-VPNs, where MACs and the information to setup flooding trees are distributed by MP-BGP . Based on , describes how to use EVPN to deliver Layer-2 services specifically in NVO3 Networks. represents a Layer-2 service deployed with an EVPN BD in an NVO3 network.

| BD1|-----+ IP1 | | |NH IP-A| | | All-Active | Hypervisor | +-------+ +----+ ESI-2 +-------------+ | +--------------------------------------+ ]]> In a simple NVO3 network, such as the example of , these are the basic constructs that EVPN uses for Layer-2 services (or Layer-2 Virtual Networks): BD1 is an EVPN Broadcast Domain for a given tenant and TS1, TS2 and TS3 are connected to it. The five represented NVEs are attached to BD1 and are connected to the same underlay IP network. That is, each NVE learns the remote NVEs' loopback addresses via underlay routing protocol. NVE1 is deployed as a virtual switch in a Hypervisor with IP-A as underlay loopback IP address. The rest of the NVEs in are physical switches and TS2/TS3 are multi-homed to them. TS1 is a virtual machine, identified by MAC1 and IP1. TS2 and TS3 are physically dual-connected to NVEs, hence they are normally not considered virtual machines.

Auto-discovery is one of the basic capabilities of EVPN. The provisioning of EVPN components in NVEs is significantly automated, simplifying the deployment of services and minimizing manual operations that are prone to human error. These are some of the Auto-Discovery and Auto-Provisioning capabilities available in EVPN: Automation on Ethernet Segments (ES): an ES is defined as a group of NVEs that are attached to the same TS or network. An ES is identified by an Ethernet Segment Identifier (ESI) in the control plane, but neither the ESI nor the NVEs that share the same ES are required to be manually provisioned in the local NVE: If the multi-homed TS or network are running protocols such as LACP (Link Aggregation Control Protocol) , MSTP (Multiple-instance Spanning Tree Protocol), G.8032, etc. and all the NVEs in the ES can listen to the protocol PDUs to uniquely identify the multi-homed TS/network, then the ESI can be "auto-sensed" or "auto-provisioned" following the guidelines in section 5. The ESI can also be auto-derived out of other parameters that are common to all NVEs attached to the same ES. As described in , EVPN can also auto-derive the BGP parameters required to advertise the presence of a local ES in the control plane (RT and RD). Local ESes are advertised using RT-4 routes and the ESI-import Route-Target used by RT-4 routes can be auto-derived based on the procedures of , section 7.6. By listening to other RT-4 routes that match the local ESI and import RT, an NVE can also auto-discover the other NVEs participating in the multi-homing for the ES. Once the NVE has auto-discovered all the NVEs attached to the same ES, the NVE can automatically perform the DF Election algorithm (which determines the NVE that will forward traffic to the multi-homed TS/network). EVPN guarantees that all the NVEs in the ES have a consistent DF Election. Auto-provisioning of services: when deploying a Layer-2 Service for a tenant in an NVO3 network, all the NVEs attached to the same subnet must be configured with a MAC-VRF and the BD for the subnet, as well as certain parameters for them. Note that, if the EVPN service model is VLAN-based or VLAN-bundle, implementations do not normally have a specific provisioning for the BD (since it is in that case the same construct as the MAC-VRF). EVPN allows auto-deriving as many MAC-VRF parameters as possible. As an example, the MAC-VRF's RT and RD for the EVPN routes may be auto-derived. Section 5.1.2.1 in specifies how to auto-derive a MAC-VRF's RT as long as VLAN-based service model is implemented. specifies how to auto-derive the RD.

Auto-discovery via MP-BGP is used to discover the remote NVEs attached to a given BD, the NVEs participating in a given redundancy group, the tunnel encapsulation types supported by an NVE, etc. In particular, when a new MAC-VRF and BD are enabled, the NVE will advertise a new RT-3 route. Besides other fields, the RT-3 route will encode the IP address of the advertising NVE, the Ethernet Tag (which is zero in case of VLAN-based and VLAN-bundle models) and also a PMSI Tunnel Attribute (PTA) that indicates the information about the intended way to deliver BUM traffic for the BD. In the example of , when BD1 is enabled, NVE1 will send an RT-3 route including its own IP address, Ethernet-Tag for BD1 and the PTA to the remote NVEs. Assuming Ingress Replication (IR) is used, the RT-3 route will include an identification for IR in the PTA and the VNI that the other NVEs in the BD must use to send BUM traffic to the advertising NVE. The other NVEs in the BD will import the RT-3 route and will add NVE1's IP address to the flooding list for BD1. Note that the RT-3 route is also sent with a BGP encapsulation attribute that indicates what NVO3 encapsulation the remote NVEs should use when sending BUM traffic to NVE1. Refer to for more information about the RT-3 route and forwarding of BUM traffic, and to for its considerations on NVO3 networks.

Tenant MAC/IP information is advertised to remote NVEs using RT-2 routes. Following the example of : In a given EVPN BD, TSes' MAC addresses are first learned at the NVE they are attached to, via data path or management plane learning. In we assume NVE1 learns MAC1/IP1 in the management plane (for instance, via Cloud Management System) since the NVE is a virtual switch. NVE2, NVE3, NVE4 and NVE5 are TOR/Leaf switches and they normally learn MAC addresses via data path. Once NVE1's BD1 learns MAC1/IP1, NVE1 advertises that information along with a VNI and Next Hop IP-A in an RT-2 route. The EVPN routes are advertised using the RD/RTs of the MAC-VRF where the BD belongs. All the NVEs in BD1 learn local MAC/IP addresses and advertise them in RT-2 routes in a similar way. The remote NVEs can then add MAC1 to their mapping table for BD1 (BT). For instance, when TS3 sends frames to NVE4 with MAC DA = MAC1, NVE4 does a MAC lookup on the BT that yields IP-A and Label L. NVE4 can then encapsulate the frame into an NVO3 tunnel with IP-A as the tunnel IP DA and L as the Virtual Network Identifier. Note that the RT-2 route may also contain the host's IP address (as in the example of ). While the MAC of the received RT-2 route is installed in the BT, the IP address may be installed in the Proxy-ARP/ND table (if enabled) or in the ARP/IP-VRF tables if the BD has an IRB. See to see more information about Proxy-ARP/ND and . for more details about IRB and Layer-3 services. Refer to and for more information about the RT-2 route and forwarding of known unicast traffic.

and are the reference documents that describe how EVPN can be used for Layer-3 services. Inter Subnet Forwarding in EVPN networks is implemented via IRB interfaces between BDs and IP-VRFs. An EVPN BD corresponds to an IP subnet. When IP packets generated in a BD are destined to a different subnet (different BD) of the same tenant, the packets are sent to the IRB attached to the local BD in the source NVE. As discussed in , depending on how the IP packets are forwarded between the ingress NVE and the egress NVE, there are two forwarding models: Asymmetric and Symmetric model. The Asymmetric model is illustrated in the example of and it requires the configuration of all the BDs of the tenant in all the NVEs attached to the same tenant. In that way, there is no need to advertise IP Prefixes between NVEs since all the NVEs are attached to all the subnets. It is called Asymmetric because the ingress and egress NVEs do not perform the same number of lookups in the data plane. In , if TS1 and TS2 are in different subnets, and TS1 sends IP packets to TS2, the following lookups are required in the data path: a MAC lookup (on BD1's table), an IP lookup (on the IP-VRF) and a MAC lookup (on BD2's table) at the ingress NVE1 and then only a MAC lookup at the egress NVE. The two IP-VRFs in are not connected by tunnels and all the connectivity between the NVEs is done based on tunnels between the BDs.

In the Symmetric model, depicted in , the same number of data path lookups is needed at the ingress and egress NVEs. For example, if TS1 sends IP packets to TS3, the following data path lookups are required: a MAC lookup at NVE1's BD1 table, an IP lookup at NVE1's IP-VRF and then IP lookup and MAC lookup at NVE2's IP-VRF and BD3 respectively. In the Symmetric model, the Inter Subnet connectivity between NVEs is done based on tunnels between the IP-VRFs.

The Symmetric model scales better than the Asymmetric model because it does not require the NVEs to be attached to all the tenant's subnets. However, it requires the use of NVO3 tunnels on the IP-VRFs and the exchange of IP Prefixes between the NVEs in the control plane. EVPN uses RT-2 and RT-5 routes for the exchange of host IP routes (in the case of the RT-2 and the RT-5 routes) and IP Prefixes (RT-5 routes) of any length. As an example, in , NVE2 needs to advertise TS3's host route and/or TS3's subnet, so that the IP lookup on NVE1's IP- VRF succeeds. specifies the use of RT-2 routes for the advertisement of host routes. Section 4.4.1 in specifies the use of RT-5 routes for the advertisement of IP Prefixes in an "Interface-less IP-VRF-to-IP-VRF Model". The Symmetric model for host routes can be implemented following either approach: uses RT-2 routes to convey the information to populate L2, ARP/ND and L3 FIB tables in the remote NVE. For instance, in , NVE2 would advertise a RT-2 route with TS3's IP and MAC addresses, and including two labels/VNIs: a label-3/VNI-3 that identifies BD3 for MAC lookup (that would be used for L2 traffic in case NVE1 was attached to BD3 too) and a label-1/VNI-1 that identifies the IP-VRF for IP lookup (and will be used for L3 traffic). NVE1 imports the RT-2 route and installs TS3's IP in the IP-VRF route table with label-1/VNI-1. Traffic from e.g., TS2 to TS3, will be encapsulated with label-1/VNI-1 and forwarded to NVE2. uses RT-2 routes to convey the information to populate the L2 FIB and ARP/ND tables, and RT-5 routes to populate the IP-VRF L3 FIB table. For instance, in , NVE2 would advertise a RT-2 route including TS3's MAC and IP addresses with a single label-3/VNI-3. In this example, this RT-2 route wouldn't be imported by NVE1 because NVE1 is not attached to BD3. In addition, NVE2 would advertise a RT-5 route with TS3's IP address and label-1/VNI-1. This RT-5 route would be imported by NVE1's IP-VRF and the host route installed in the L3 FIB associated to label-1/VNI-1. Traffic from TS2 to TS3 would be encapsulated with label-1/VNI-1.

describes how to use EVPN for NVO3 encapsulations, such us VXLAN, nvGRE or MPLSoGRE. The procedures can be easily applicable to any other NVO3 encapsulation, in particular GENEVE. The Generic Network Virtualization Encapsulation has been recommended to be the proposed standard for NVO3 Encapsulation. The EVPN control plane can signal the GENEVE encapsulation type in the BGP Tunnel Encapsulation Extended Community (see ). The NVO3 encapsulation design team has made a recommendation in for a control plane to: Negotiate a subset of GENEVE option TLVs that can be carried on a GENEVE tunnel Enforce an order for GENEVE option TLVs and Limit the total number of options that could be carried on a GENEVE tunnel. The EVPN control plane can easily extend the BGP Tunnel Encapsulation Attribute sub-TLV to specify the GENEVE tunnel options that can be received or transmitted over a GENEVE tunnels by a given NVE. describes the EVPN control plane extensions to support GENEVE.

EVPN OAM (as in ) defines mechanisms to detect data plane failures in an EVPN deployment over an MPLS network. These mechanisms detect failures related to P2P and P2MP connectivity, for multi-tenant unicast and multicast L2 traffic, between multi-tenant access nodes connected to EVPN PE(s), and in a single-homed, single-active or all-active redundancy model. In general, EVPN OAM mechanisms defined for EVPN deployed in MPLS networks are equally applicable for EVPN in NVO3 networks.

EVPN can be used to signal the security protection capabilities of a sender NVE, as well as what portion of an NVO3 packet (taking a GENEVE packet as an example) can be protected by the sender NVE, to ensure the privacy and integrity of tenant traffic carried over the NVO3 tunnels .

This section describes how EVPN can be used to deliver advanced capabilities in NVO3 networks.

replaces the traditional Ethernet Flood-and-Learn behavior among NVEs with BGP-based MAC learning, which in return provides more control over the location of MAC addresses in the BD and consequently advanced features, such as MAC Mobility. If we assume that VM Mobility means the VM's MAC and IP addresses move with the VM, EVPN's MAC Mobility is the required procedure that facilitates VM Mobility. According to section 15, when a MAC is advertised for the first time in a BD, all the NVEs attached to the BD will store Sequence Number zero for that MAC. When the MAC "moves" within the same BD but to a remote NVE, the NVE that just learned locally the MAC, increases the Sequence Number in the RT-2 route's MAC Mobility extended community to indicate that it owns the MAC now. That makes all the NVE in the BD change their tables immediately with no need to wait for any aging timer. EVPN guarantees a fast MAC Mobility without flooding or black-holes in the BD.

The advertisement of MACs in the control plane, allows advanced features such as MAC protection, Duplication Detection and Loop Protection. MAC Protection refers to EVPN's ability to indicate - in a RT-2 route - that a MAC must be protected by the NVE receiving the route. The Protection is indicated in the "Sticky bit" of the MAC Mobility extended community sent along the RT-2 route for a MAC. NVEs' ACs that are connected to subject-to-be-protected servers or VMs, may set the Sticky bit on the RT-2 routes sent for the MACs associated to the ACs. Also, statically configured MAC addresses should be advertised as Protected MAC addresses, since they are not subject to MAC Mobility procedures. MAC Duplication Detection refers to EVPN's ability to detect duplicate MAC addresses. A "MAC move" is a relearn event that happens at an access AC or through a RT-2 route with a Sequence Number that is higher than the stored one for the MAC. When a MAC moves a number of times N within an M-second window between two NVEs, the MAC is declared as Duplicate and the detecting NVE does not re-advertise the MAC anymore. provides MAC Duplication Detection, and with an extension it can protect the BD against loops created by backdoor links between NVEs. The same principle (based on the Sequence Number) may be extended to protect the BD against loops. When a MAC is detected as duplicate, the NVE may install it as a black-hole MAC and drop received frames with MAC SA and MAC DA matching that duplicate MAC. The MAC Duplication extension to support Loop Protection is described in .

In BDs with a significant amount of flooding due to Unknown unicast and Broadcast frames, EVPN may help reduce and sometimes even suppress the flooding. In BDs where most of the Broadcast traffic is caused by ARP (Address Resolution Protocol) and ND (Neighbor Discovery) protocols on the TSes, EVPN's Proxy-ARP and Proxy-ND capabilities may reduce the flooding drastically. The use of Proxy-ARP/ND is specified in . Proxy-ARP/ND procedures along with the assumption that TSes always issue a GARP (Gratuitous ARP) or an unsolicited Neighbor Advertisement message when they come up in the BD, may drastically reduce the unknown unicast flooding in the BD. The flooding caused by TSes' IGMP/MLD or PIM messages in the BD may also be suppressed by the use of IGMP/MLD and PIM Proxy functions, as specified in and . These two documents also specify how to forward IP multicast traffic efficiently within the same BD, translate soft state IGMP/MLD/PIM messages into hard state BGP routes and provide fast-convergence redundancy for IP Multicast on multi-homed Ethernet Segments (ESes).

When an NVE attached to a given BD needs to send BUM traffic for the BD to the remote NVEs attached to the same BD, Ingress Replication is a very common option in NVO3 networks, since it is completely independent of the multicast capabilities of the underlay network. Also, if the optimization procedures to reduce/suppress the flooding in the BD are enabled (), in spite of creating multiple copies of the same frame at the ingress NVE, Ingress Replication may be good enough. However, in BDs where Multicast (or Broadcast) traffic is significant, Ingress Replication may be very inefficient and cause performance issues on virtual-switch-based NVEs. specifies the use of AR (Assisted Replication) NVO3 tunnels in EVPN BDs. AR retains the independence of the underlay network while providing a way to forward Broadcast and Multicast traffic efficiently. AR uses AR-REPLICATORs that can replicate the Broadcast/Multicast traffic on behalf of the AR-LEAF NVEs. The AR-LEAF NVEs are typically virtual-switches or NVEs with limited replication capabilities. AR can work in a single-stage replication mode (Non-Selective Mode) or in a dual-stage replication mode (Selective Mode). Both modes are detailed in . In addition, also describes a procedure to avoid sending Broadcast, Multicast or Unknown unicast to certain NVEs that do not need that type of traffic. This is done by enabling PFL (Pruned Flood Lists) on a given BD. For instance, an virtual-switch NVE that learns all its local MAC addresses for a BD via Cloud Management System, does not need to receive the BD's Unknown unicast traffic. Pruned Flood Lists help optimize the BUM flooding in the BD.

Another fundamental concept in EVPN is multi-homing. A given TS can be multi-homed to two or more NVEs for a given BD, and the set of links connected to the same TS is defined as Ethernet Segment (ES). EVPN supports single-active and all-active multi-homing. In single-active multi-homing only one link in the ES is active. In all-active multi-homing all the links in the ES are active for unicast traffic. Both modes support load-balancing: Single-active multi-homing means per-service load-balancing to/from the TS. For example, in , for BD1, only one of the NVEs can forward traffic from/to TS2. For a different BD, the other NVE may forward traffic. All-active multi-homing means per-flow load-balanding for unicast frames to/from the TS. That is, in and for BD1, both NVE4 and NVE5 can forward known unicast traffic to/from TS3. For BUM traffic only one of the two NVEs can forward traffic to TS3, and both can forward traffic from TS3. There are two key aspects in the EVPN multi-homing procedures: DF (Designated Forwarder) election: the DF is the NVE that forwards the traffic to the ES in single-active mode. In case of all-active, the DF is the NVE that forwards the BUM traffic to the ES. Split-horizon function: prevents the TS from receiving echoed BUM frames that the TS itself sent to the ES. This is especially relevant in all-active ESes, where the TS may forward BUM frames to a non-DF NVE that can flood the BUM frames back to the DF NVE and then the TS. As an example, in , assuming NVE4 is the DF for ES-2 in BD1, BUM frames sent from TS3 to NVE5 will be received at NVE4 and, since NVE4 is the DF for DB1, it will forward them back to TS3. Split-horizon allows NVE4 (and any multi-homed NVE for that matter) to identify if an EVPN BUM frame is coming from the same ES or different, and if the frame belongs to the same ES2, NVE4 will not forward the BUM frame to TS3, in spite of being the DF. While describes the default algorithm for the DF Election, and specify other algorithms and procedures that optimize the DF Election. The Split-horizon function is specified in and it is carried out by using a special ESI-label that it identifies in the data path, all the BUM frames being originated from a given NVE and ES. Since the ESI-label is an MPLS label, it cannot be used in all the non-MPLS NVO3 encapsulations, therefore defines a modified Split-horizon procedure that is based on the IP SA of the NVO3 tunnel, and it is known as "Local-Bias". It is worth noting that Local-Bias only works for all-active multi-homing, and not for single-active multi-homing.

describes how EVPN can be used for Inter Subnet Forwarding among subnets of the same tenant. RT-2 routes and RT-5 routes allow the advertisement of host routes and IP Prefixes (RT-5 route) of any length. The procedures outlined by are similar to the ones in , only for NVO3 tunnels. However, also defines advanced Inter Subnet Forwarding procedures that allow the resolution of RT-5 routes to not only BGP next-hops but also "overlay indexes" that can be a MAC, a GW IP or an ESI, all of them in the tenant space. illustrates an example that uses Recursive Resolution to a GW-IP as per section 4.4.2. In this example, IP-VRFs in NVE1 and NVE2 are connected by a SBD (Supplementary BD). An SBD is a BD that connects all the IP-VRFs of the same tenant, via IRB, and has no ACs. NVE1 advertises the host route TS2-IP/L (IP address and Prefix Length of TS2) in an RT-5 route with overlay index GWIP=IP1. Also, IP1 is advertised in an RT-2 route associated to M1, VNI-S and BGP next-hop NVE1. Upon importing the two routes, NVE2 installs TS2-IP/L in the IP-VRF with a next-hop that is the GWIP IP1. NVE2 also installs M1 in the SBD, with VNI-S and NVE1 as next-hop. If TS3 sends a packet with IP DA=TS2, NVE2 will perform a Recursive Resolution of the RT-5 route prefix information to the forwarding information of the correlated RT-2 route. The RT-5 route's Recursive Resolution has several advantages such as better convergence in scaled networks (since multiple RT-5 routes can be invalidated with a single withdrawal of the overlay index route) or the ability to advertise multiple RT-5 routes from an overlay index that can move or change dynamically. describes a few use-cases.

| | RT-5(TS2-IP/L,GWIP=IP1)--> | +-------------------------------------+ ]]>

The concept of the SBD described in is also used in for the procedures related to Inter Subnet Multicast Forwarding across BDs of the same tenant. For instance, allows the efficient forwarding of IP multicast traffic from any BD to any other BD (or even to the same BD where the Source resides). The procedures are supported along with EVPN multi-homing, and for any tree allowed on NVO3 networks, including IR or AR. also describes the interoperability between EVPN and other multicast technologies such as MVPN (Multicast VPN) and PIM for inter-subnet multicast. describes another potential solution to support EVPN to MVPN interoperability.

Tenant Layer-2 and Layer-3 services deployed on NVO3 networks must be extended to remote NVO3 networks that are connected via non-NOV3 WAN networks (mostly MPLS based WAN networks). defines some architectural models that can be used to interconnect NVO3 networks via MPLS WAN networks. When NVO3 networks are connected by MPLS WAN networks, specifies how EVPN can be used end-to-end, in spite of using a different encapsulation in the WAN. also supports the use of NVO3 or Segment Routing (encoding 32-bit or 128-bit Segment Identifiers into labels or IPv6 addresses respectively) transport tunnels in the WAN. Even if EVPN can also be used in the WAN for Layer-2 and Layer-3 services, there may be a need to provide a Gateway function between EVPN for NVO3 encapsulations and IPVPN for MPLS tunnels, if the operator uses IPVPN in the WAN. specifies the interworking function between EVPN and IPVPN for unicast Inter Subnet Forwarding. If Inter Subnet Multicast Forwarding is also needed across an IPVPN WAN, describes the required interworking between EVPN and MVPN (Multicast Virtual Private Networks).