End Marker in 5G Data Network

Introduction In a mobile network like 5G , when a mobile device (referred to as User Equipment or UE) moves from one Access Node (source AN) to another (target AN), the anchoring node User Plane Function (UPF) that connects the UE to the Data Network sends an End Marker to the source AN after it starts sending UE traffic towards the target AN. The target AN buffers the data until it receives the End Marker forwarded by the source AN. This is to preserve the order of packets during the handover between ANs.

End Marker with Centralized UPF The End Marker mechanism in case of centralized UPF is described with the following diagram.

| \ / UE1 ]]> UE1 is initially connected to AN1. There is an N3 tunnel between AN1 and UPF for the UE. The UE1-AN1 connection and the AN1-UPF N3 tunnel together provide a Packet Data Unit Session (PDU Session) that connects UE1 to the Data Network (DN, which is Internet in this case) via the UPF. The UPF does IP routing between the DN and UE (among other things). When UE1 moves from AN1 (source) to AN2 (target), a new N3 tunnel between AN2 and UPF is established as instructed by 5G control plane functions. For Down Link (DL) traffic from the Internet, UPF starts sending towards AN2 via the new N3 tunnel. There could be inflight packets towards AN1, which will be redirected by AN1 to AN2. The redirected packets were sent by the UPF before it switches over to the new tunnel, yet they could arrive on AN2 after the packets that the UPF sends over the new tunnel to AN2. To preserve the packet ordering during this handover, the following sequence of actions are taken: The UPF sends an End Marker to the source AN1 over the old tunnel after it sends the last packet over the old tunnel and switches to the new tunnel for future packets. The source AN1 redirects inflight packets to the target AN2, who forwards the redirected traffic to UE1, while buffering data arriving on the new tunnel. The source AN1 redirects the received End Marker (which comes after the inflight packets) to the target AN2, who will now start sending buffered and new arrival data to UE1.

End Marker with Distributed UPF/ANUP With distributed UPF, the diagram becomes the following:

| \ / UE1 ]]> UPF1 and UPF2 are close to the ANs. The DN is an IP VPN (referred to as DNVPN) provided by PE1/PE2 and a hub PE (where the previous central UPF was) connected to the Internet. The UPFs are VPN CEs. proposes an Integrated AN/UPF (ANUP) concept for co-located AN and UPFs. Whether AN and UPFs are separate or integrated, the problem discussion and proposed solution in this document apply. As described in , when UE1 moves from AN1/UPF1 to AN2/UPF2, its IP address may be retained if that is desired. A UPF has UE specific routes/state for its local UEs so that DL traffic can be forwarded accordingly. The hub PE has UE specific routes for all UEs that have persistent addresses so that DL traffic may be sent to the appropriate UPFs. While some PEs may also have UE specific routes for UEs attached to other UPFs, some may just have a default route to send all Up Link (UL) traffic (traffic not destined to its local UEs) towards the hub PE. This is achieved by standard hub-and-spoke VPN mechanisms: some routes are advertised with Route Targets that allow them to be installed everywhere, while some other routes are advertised with different Route Targets that allow them to be installed only on hub routers.

Traffic via a Hub Router For DL traffic from the Internet via the hub router, re-ordering may happen when UE1 moves from AN1/UPF1 to AN2/UPF2 - after the hub PE updates its UE1 route to send traffic to UPF2, there may be inflight UE1 packets towards PE/UPF1 but then rerouted to PE2/UPF2 (either directly or via the hub PE) in the DN, and they may arrive after the traffic that the hub sends to UPF2 directly. Notice that during this process, UPF1 will not trigger an End Marker to AN1 because there is no N3 tunnel changing on UPF1. Instead, the hub PE needs to trigger UPF1 to send an End Marker to AN1 after the hub router changes its UE1 route from PE1 to PE2. Because PEs are just DN devices not 5G function, an ICMP message can be sent instead by the hub PE towards UPF1's address in the DN. The ICMP message is of a new type/code and carries the UE route so that UPF1 can match the PDU session and send End Marker to AN1. In certain cases, e.g., with , the hub PE may know the PDU session information associated with a UE route and it can include the session information in the ICMP message. Note that, if PE1 also maintains UE routes received from other PEs, it prefers its own received from UPF1. This is typical IPVPN behavior that local routes are preferred. This ensures that inflight traffic (before the ICMP trigger for End Marker) from the hub PE is sent to UPF1 instead of routed to PE2/UPF2. Otherwise, reordering may happen (inflight traffic before the End Markter trigger arrives after the traffic that is sent to UPF2 after the trigger). After UPF1 sends End Marker, it withdraws the UE1 route from PE1.

Traffic from Anywhere In the previous section, only traffic from the hub PE is considered. However, with distributed UPFs, DL traffic may be from anywhere directly without going through a hub. For example, there could be a UPF3/PE3 connecting to a local DN site and traffic from that site via PE3 or UE traffic via UPF3 to UE1 also needs to preserve order when UE1 moves to UPF2/AN2. If PE3 has UE1 specific route (previously through PE1/UPF1 then through PE2/UPF2 directly instead of just a default route via the hub PE), then UPF3 also needs to trigger UPF1 to send End Marker. In this case, the source UPF1 needs to know how many triggers it needs to receive before sending the End Marker. The number is dependent on how many PEs are configured with the Route Target that the UE1 route is advertised with (those PEs will install the UE1 router and send End Marker trigger when their UE1 route changes from PE1/UPF1 to PE2/UPF2). Since the trigger could be lost, and some PEs may be down or slow in sending the trigger, UPF1 needs to send End Marker to its AN when one of the following conditions are met: The expected number of triggers are received, or, A preset timer expires Alternatively, the source UPF1 can send the End Marker upon receipt of a master trigger from a controller.

Remote UE Routes on Source UPF In both and scenarios, the source PE1 may have the specific UE1 route that is advertised from the target PE2. After the hub PE or PE3 switches to the target PE2/UPF2, inflight traffic sent before the switch, arriving on PE1 and then following the UE1 route on PE1, may arrive on UPF2 later than the traffic sent to UPF2 directly after the switch, if on PE1 the UE1 route through PE2/UPF2 is preferred over the UE1 route through UPF1. With VPN the CE routes through a PE-CE interface are typically preferred over those learned from another PE - so this should work just fine

Is End Marker Mechanism Really Needed? One may wonder if the it is really necessary to extend the End Marker mechanism to the DN, given the following considerations: Complexity especially when traffic is not just from a hub PE. With higher speed of 5G/6G, buffering data may become more and more difficult. IP network does not guarantee packet ordering. While routers do try to send traffic of the same flow out of the same ECMP branch, topology change will likely cause reordering and routers do not buffer data for ordering purposes. On the other hand, mobile network is unique in that mobility handover happens more frequently than topology changes in traditional IP networks, and in the discussions related to session continuity in the context of distributed UPF and ANUP, preserving packet ordering is often brought up (at least rhetorically). Therefore, this document does discuss the problem and propose a solution in case it is indeed needed.