Mobility Procedures in Presence of Unknown MAC Route

Ethernet Virtual Private Network (EVPN) solution is becoming pervasive for Network Virtualization Overlay (NVO) services in data center (DC) and Enterprise applications and as the next generation Virtual Private Network (VPN) services in service provider (SP) applications. The host IP default route and host unknown MAC route within a DC can be used in order to ensure that leaf nodes within a DC only learn and store host MAC and IP addresses for that DC. All other host MAC and IP addresses from remote DCs are learned and stored in DC GW nodes thus alleviating leaf nodes from learning host MAC and IP addresses from the remote DCs and potentially improving the scale of MAC and IP addresses on leaf nodes by one to two order of magnitude. Interconnect Solution for Ethernet VPN defines Unknown MAC Route (UMR) utilization for Data Center Interconnect (DCI) when EVPN MPLS or EVPN VXLAN is used as overlay network for such interconnects. The introduction of UMR for such scenarios impacts MAC mobility procedures that are not discussed in . This document describes additional changes and enhancements needed for MAC mobility procedures when UMR is used. ("") of this document discusses the requirements for MAC Mobility when UMR is used. ("") discusses MAC Mobility for DCI operation without UMR utilization. Section 5 discusses MAC Mobility for DCI operation with UMR and the modifications and enhancements needed on top of the baseline operation discussed in section 4.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

EVI:: An EVPN instance spanning the Provider Edge (PE) devices participating in that EVPN. An EVI may be comprised of one BD (VLAN-based, VLAN Bundle, or Port-based services) or multiple BDs (VLAN-aware Bundle or Port-based VLAN-Aware services).
MAC-VRF:: A Virtual Routing and Forwarding table for Media Access Control (MAC) addresses on a PE.
Ethernet Segment (ES):: When a customer site (device or network) is connected to one or more PEs via a set of Ethernet links, then that set of links is referred to as an 'Ethernet segment'.
Ethernet Segment Identifier (ESI):: A unique non-zero identifier that identifies an Ethernet segment is called an 'Ethernet Segment Identifier'.
VID:: VLAN Identifier.
Ethernet Tag:: Used to represent a BD that is configured on a given ES for the purposes of DF election and <EVI, BD> identification for frames received from the CE. Note that any of the following may be used to represent a BD: VIDs (including Q-in-Q tags), configured IDs, VNIs (Virtual Extensible Local Area Network (VXLAN) Network Identifiers), normalized VIDs, I-SIDs (Service Instance Identifiers), etc., as long as the representation of the BDs is configured consistently across the multihomed PEs attached to that ES.
Ethernet Tag ID:: Normalized network wide ID that is used to identify a BD within an EVI and carried in EVPN routes.
MP2MP:: Multipoint to Multipoint.
MP2P:: Multipoint to Point.
P2MP:: Point to Multipoint.
P2P:: Point to Point.
PE:: Provider Edge device.
Single-Active Redundancy Mode:: When only a single PE, among all the PEs attached to an Ethernet segment, is allowed to forward traffic to/from that Ethernet segment for a given VLAN, then the Ethernet segment is defined to be operating in Single-Active redundancy mode.
All-Active Redundancy Mode:: When all PEs attached to an Ethernet segment are allowed to forward known unicast traffic to/from that Ethernet segment for a given VLAN, then the Ethernet segment is defined to be operating in All-Active redundancy mode.
BUM:: Broadcast, unknown unicast, and multicast.
DF:: Designated Forwarder.
Backup-DF (BDF):: Backup-Designated Forwarder.
Non-DF (NDF):: Non-Designated Forwarder.
AC:: Attachment Circuit.
NVO:: Network Virtualization Overlay as described in
IRB:: Integrated Routing and Bridging interface, with EVPN procedures described in

This section lists the requirements for the MAC mobility solution when UMR is used.

When an EVPN overlay network (i.e., DC, Enterprise, or SP network) is enabled for UMR operation, then all the PEs (leaf nodes) and all the gateways (border leaf nodes) in that network MUST support UMR operation - e.g., a DC network cannot have some leaf nodes supporting UMR operation and some other leaf nodes incapable of UMR operation. This means when upgrading a DC network for UMR capability, all leaf nodes (PEs), all border leaf nodes (gateways), and all Route Reflectors (RRs) in that network needs to be upgraded before turning on the UMR capability. If desired, a physical DC network can be partitioned into two logical networks with one supporting UMR operation and the other not supporting it.
UMR MAC mobility procedures for DC networks that are UMR capable MUST operate seamlessly with DC networks that are UMR incapable.
UMR operation is optional and a PE device (a leaf node) that supports UMR procedure but doesn't receive the UMR route from its gateway (its border leaf), SHALL operate per baseline . This does not mean that a DC network can partially operate in UMR mode but rather it means that a DC network can gradually be upgraded for UMR capability and once the entire network is upgraded, then it can operate in UMR mode.

In order to better differentiate the enhancements needed to MAC Mobility procedures for networks interconnect scenarios with utilization of UMR which is missing in , we first start with the description of the baseline MAC Mobility procedures (without utilization of UMR) in this section and then we describe the changes needed on top of this baseline scenario in the next section. The following ladder diagram is used to help in describing baseline MAC Mobility operation.

| |AD(M1) |AD(M1) | | |---------|-------->|---------------->|-----------|--------->| | | | |---------->| | | | | | | | | AD(M1,1)|AD(M1,1) |AD(M1,1) |AD(M1,1) | | 2) |<--------X-------->|---------------->|-----------|--------->| | | | |---------->| | |WD(M1) | | | | | |-------->| | | | | |---------|-------->| | | | | | | | | | | | | | | AD(M1,2)X | | AD(M1,2)| AD(M1,2)| AD(M1,2)|<---------| | |<--------|<----------------|<----------|----------| |<--------|---------| | | | 3) | | | | | | | WD(M1)|WD(M1) |WD(M1) |WD(M1) | | |<--------|-------->|---------------->|-----------|--------->| | | | |---------->| | | | | | | | | | | | | | | | AD(M1,3)| AD(M1,3)| AD(M1,3)|AD(M1,3) | 4) | |<--------|<----------------|<----------X--------->| |<--------|---------| | | WD(M1)| | | | | |<---------| | | | |<----------|----------| AD(M1,x) = MAC/IP Route Advertisement for MAC M1 with seq# x WD(M1) = MAC/IP Route Withdrawl for MAC M1 X = Host M1 being local to that PE ]]> This diagram depicts MAC Mobility procedures between two overlay networks which in turn are connected via a WAN network; where, GW1 and GW2 sit at the edge of the WAN. The first network consists of PE11, PE12, and GW1. The second network consists of PE21, PE22, and GW2. EVPN control plane is used both within each network as well as between them.

PE11 learns host M1 for the first time and it advertises it via MAC/IP Route Advertisement to other nodes in its DC network participating in that EVI. Since this is the first time that PE11 learns of M1, it sends the advertisement without MAC Mobility extended community attribute per section 15 of . When the local gateway (GW1 of DC1) receives this advertisement, it readvertises this MAC address per procedures of without MAC Mobility extended community attribute. The remote gateway (GW2 of DC2) receives this advertisement and it in turn readvertises it to its PEs participating in that EVI. At this point, remote PE21 and PE22 know that the next hop for M1 is GW2, and GW2 knows that the next hop for M1 is GW1.
In this step, host M1 makes an intra-DC move within DC1 network and moves from PE11 to PE12, the PE12 follows the MAC mobility procedures in and advertises the MAC/IP Advertisement route for M1 with a sequence number which is incremented by one (in this case seq = 1). PE11 and GW1 receive this advertisement and update their next hop for M1 to point to PE12. GW1 follows the procedures in and it readvertises this route with this new sequence number received from PE12. Upon receiving this route, GW2 updates its sequence number for M1 and in turn it readvertises this route to its DC. PE21 and PE22 receive this advertisement and update their sequence numbers for M1 (seq = 1), but there is no change to the next hop for M1 and they keep it as GW2. Furthermore, after verifying that M1 is no longer present locally, PE11 sends a withdrawal message for M1 to all local PEs and GWs that are participating in that EVI per MAC Mobility procedures of . When PE12 and GW1 receive this withdrawal message, they clean up their BGP tables and remove BGP EVPN route for M1 received from PE11. Since BGP table in GW1 has at least one BGP EVPN route learned from its DC1 (and host M1 has as its next hop one of the PEs in DC1), GW1 does not readvertise the withdrawal message for M1 received from PE11.
In this step, host M1 moves from PE12 in DC1 to PE21 in DC2. PE21 upon learning M1 locally, it advertises the MAC/IP Advertisement route for M1 with a sequence number which is incremented by one (in this case seq = 2). PE22 and GW2 receive this advertisement, recognize the move, and update their next hop for M1 to point to PE21. GW2 follows the procedures in and it readvertises this route with this new sequence number received from PE21. Upon receiving this route, GW1 updates its sequence number for M1 and in turn it readvertises this route to its DC1 network. PE11 and PE12 receive this advertisement, recognize the move, and update their next hop to point to GW1, and their sequence numbers for M1. Furthermore, after verifying that M1 is no longer present locally, PE12 sends a withdrawal message for M1 to all local PEs and GWs that are participating in that EVI. When PE11 and GW1 receive this withdrawal message, they clean up their BGP tables and remove BGP EVPN route for M1 received from PE12. Since there is no more local BGP EVPN routes for M1 in BGP table of GW1 (i.e., no more routes from its local PEs), it readvertises this withdrawal message to other GWs over WAN. When other GWs over WAN (including GW2) receive this withdrawal message, they remove the BGP EVPN route for M1 received from GW1. At this point the only BGP EVPN route entry in GW1 is the one received from GW2, and for GW2 is the one received from its local PE21.
This step is similar to that of step 2 and demonstrates what it takes place when an intra-DC move happens but this time within DC2 where the host M1 moves from PE21 to PE22. The PE22 follows the MAC mobility procedures in and advertises the MAC/IP Advertisement route for M1 with a sequence number which is incremented by one (in this case seq = 3). PE21 and GW2 receive this advertisement and update their next hop for M1 to point to PE22. GW2 follows the procedures in and it readvertises this route with this new sequence number received from PE22. Upon receiving this route, GW1 updates its sequence number for M1 and in turn it readvertises this route to its DC. PE11 and PE12 receive this advertisement and update their sequence numbers for M1 (seq = 3), but there is no change to the next hop for M1 and they keep it as GW1. Furthermore, after verifying that M1 is no longer present locally, PE21 sends a withdrawal message for M1 to all local PEs and GWs that are participating in that EVI per MAC Mobility procedures of . When PE22 and GW2 receive this withdrawal message, they clean up their BGP tables and remove BGP EVPN route for M1 received from PE21. Since BGP table in GW2 has at least one BGP EVPN route learned from its DC2 (and host M1 has as its next hop one of the PEs in DC2), GW2 does not readvertise the withdrawal message for M1 received from PE21.

This section describes the changes needed to MAC Mobility procedures when UMR is utilized in EVPN overlay networks and hosts are allowed to move between these networks. Since advertisement of MAC/IP addresses from the local overlay network are not propagated all the way to the PEs of the remote overlay network, the baseline MAC mobility procedures described in the previous section, cannot be used as is and it needs to be modified. When a host M1 moves from one overlay network (e.g., DC1) to anoher one (e.g., DC2), the PE in DC2 (PE21) that learns the host locally, it learns it for the very first time because of UMR operation - i.e., M1 never previously got advertised by DC2 GW (GW2). Therefore, the PE21 advertises EVPN MAC/IP route for M1 without any sequence number which breaks the baseline MAC mobility procedures described in the previous section. In order to accommodate MAC mobility in the presence of UMR, each gateway needs to maintain two sequence numbers per host MAC address - one for its local overlay network (e.g., its DC network) and another one for the interconnect network (e.g., WAN network). Only gateways need to maintain both MAC Mobility sequence numbers. The PEs that are enabled for UMR operation, only need to maintain a single MAC Mobility sequence number per MAC address. These two sequence numbers operate independently (i.e., they get incremented independently) so that a local MAC move within an overlay network (e.g., DC1) does not impact other overlay networks (e.g., other DCs) and the interconnect network (e.g., WAN network) - i.e., when the host mobility is confined to a DC (aka intra-DC host mobility), then only the intra-DC MAC Mobility counter for that DC is incremented upon host move without any changes to inter-DC MAC Mobility counter or any other intra-DC MAC Mobility counters in other DCs. However, when the host moves from one DC to another, then the inter-DC MAC Mobility counter is impacted. The solution described in this section optimizes based on convergence time and number of BGP EVPN route advertisements - i.e., it tries to minimize the convergence time upon a host move and to minimize the number of EVPN route advertisements. Whenever these two factors are in conflict, the preference is given to minimizing the convergence time. The following ladder diagram is used to help in describing MAC Mobility procedures for UMR operation.

| |AD(M1) | | | |---------|-------->|---------------->| | | | | | | | | | | | | | | | AD(M1,1)|AD(M1,1) | | | | |<--------X-------->| | | | | | | | | | 2) |WD(M1) | | | | | |-------->| | | | | |---------|-------->| | | | | | | | | | | | | | | AD(M1)X | | AD(M1,2)| AD(M1,1)| AD(M1)|<---------| | |<--------|<----------------|<----------|----------| |<--------|---------| | | | | | | | | | | WD(M1)|WD(M1) | | | | 3) |<--------|-------->|WD(M1) | | | | |<--------|---------------->| | | |<--------|---------| | | | | | | | AD(M1,1)|AD(M1,1) | | | | |<----------X--------->| | | | | | | 4) | | | | | WD(M1)| | | | | |<---------| | | | |<----------|----------| AD(M1,x) = MAC/IP Route Advertisement for MAC M1 with seq# x WD(M1) = MAC/IP Route Withdrawl for MAC M1 X = Host M1 being local to that PE ]]> This diagram depicts MAC Mobility procedures between two overlay networks which in turn are connected via a WAN network where UMR operation is utilized. To avoid repeating the text verbatim from previous section and put the emphasis on the new procedures, we mainly elaborate on the changes relative to the baseline MAC mobility procedures described in the previous section. When UMR operation is enabled for a given EVI, all gateways participating in that EVI for that overlay network, advertise the UMR route to their local overlay network. A PE that is capable of UMR processing, upon receiving the UMR route, activates its UMR procedure as described below. When a gateway receives the UMR route from another gateway for one of its EVI for which UMR operation is enabled, it should simply discard it (i.e., not to add it to its BGP table and MAC-VRF).

When the host M1 is learned in DC1 for the first time, the baseline MAC Mobility procedure described in step (1) of is executed in DC1 among PE11, PE12, and GW1. When the remote gateway (GW2 of DC2) receives this advertisement from GW1, it processes it just as step (1) of and adds it to its BGP table and MAC-VRF for that EVI. However, it does not readvertise it into its own DC network because it is configured for UMR operation and no remote MAC/IP Advertisement routes (routes received from remote GWs) are ever readvertised locally.
When the host M1 makes an intra-DC move within DC1 network, the baseline MAC Mobility procedure described in step (2) of is executed in DC1 among PE11, PE12, and GW1. GW1 realizes that this is an intra-overlay network MAC move (intra-DC MAC move) and thus it does not readvertises this route to other GWs in the WAN network. It should be noted that GW1 maintains two sequence numbers for M1 and it increments its intra-DC sequence number by one (seq = 1); however, it leaves its inter-DC sequence number unchanged (seq = 0). Generation of the withdrawal message by PE11 and processing of this message by other EVPN devices in DC1 (i.e., PE12 and GW1) are the same as the ones described in step (2) of . Since after receiving the withdrawal message and cleaning up its BGP table, GW1 still has at least one BGP EVPN route from its local DC1 (and host M1 has as its next hop one of the PEs in DC1), GW1 does not readvertise the withdrawal message for M1 received from PE11 to remote gateways (e.g., GW2 of DC2).
When the host M1 moves from DC1 to DC2 and its presence is detected locally by the PE21, the PE21 learns M1 for the very first time since its gateway (GW2) never advertised MAC/IP route for M1 to its local PEs (because of UMR operation). The PE21 advertises MAC/IP route for M1 without any sequence number. All PEs and gateways in DC2 upon receiving this advertisement update their BGP and MAC-VRF tables. In addition to this update, the GW2 recognizes that there has been a MAC move, increments its inter-DC MAC Mobility counter for M1, and it readvertises this MAC/IP route along with the updated MAC Mobility extended community. GW1 receives this MAC/IP advertisement for M1 and it also recognizes that M1 has moved to DC2. GW1 increments its intra-DC MAC Mobility sequence number and it readvertises this route along with the updated MAC Mobility extended community (seq = 2) to its local DC for that EVI. As the result, all the PEs participating for that EVI in DC1, receive this MAC/IP advertisement and update their BGP and MAC-VRF tables. They also update the BGP next hop to point to GW1 for MAC address M1. Besides this update, PE12 recognizes the MAC move and advertises a withdrawal message for M1. Furthermore, it verifies that M1 has actually moved and is no longer present locally. When the gateways and other PEs in DC1 receive this withdrawal message from PE12, they cleanup their BGP tables and remove the corresponding M1 entry from their tables. After this cleanup, GW1 realizes that there is no more entry for M1 in its BGP table from its local PEs and thus it sends a withdrawal message for M1 to all its local PEs and remote gateways (e.g., GW2). Furthermore, GW1 must reset its intra-DC MAC mobility counter for M1 to zero because M1 no longer exist among its local PEs. When the local PEs (PE11 and PE12) receive this withdrawal message, they clean up their BGP and MAC-VRF tables for M1. After the cleanup, there should be no entry in BGP and MAC-VRF tables for M1 and thus the forwarding for M1 must follow UMR operation - i.e., the packet with the destination MAC address of M1 must be load balanced to one of the GWs that has advertised UMR route. When the remote gateways receive this withdrawal message, they clean up their BGP tables for M1 and the only entry in BGP table for M1 should be that of the one received from GW2.
This step demonstrates an intra-DC MAC move for DC2. The procedure for the PEs (PE21 and PE22) and the corresponding gateway (GW2) is the same as the one described in step 2 and thus no further explanation is needed.

Redundant gateways are supported by the described procedure. All the redundant gateways attached to a given BD advertise the EVPN MAC/IP Advertisement routes with the same Interconnect ESI [RFC9014], and all the redundant gateways MUST use the same sequence numbers when advertising MAC addresses to either their local overlay network or their interconnect network.

Duplicate MAC addresses can occur as described in section 15.1 of . MAC address duplication can happen within the same DC network (e.g., DC1) or across different DC networks (e.g., DC1 and DC2) where UMR is utilized. In either case, the procedure is the same as the one described in section 15.1 of . More specifically, the timer and the move counter for a given MAC are kept only at the PEs - i.e., there is no need to maintain such timer and move counter for a given MAC unless that MAC is learned locally on that GW.

This section describes MAC Mobility procedures for hosts sitting behind local Attachment Circuts (ACs) of a gateway and moving to/from a local PE, or another gateway in a remote DC, or a remote PE in a remote DC. TBD.

Since this document describes how to address MAC mobility issue as the result of using UMR for interconnection solutions of , and since no new requiremnts with respect to mobility procedures are introduced at the edge devices (e.g., PEs or leafs), there is no additional security risks beyond the ones described in and .

This document does not introduce any IANA requirements.