Considerations for Benchmarking Network Performance in Satellite Internet Constellations

Introduction In the past few years, thanks to the innovative technologies emerged from the aerospace industry, we have witnessed the rapid evolution and deployment of satellite Internet constellations (SIC) in low earth orbit (LEO). These SICs, such as SpaceX's Starlink, OneWeb and Amazon's Kuiper project, are actively deploying hundreds to thousands of broadband LEO satellites in the outer space, and they promise to realize pervasive, high-throughput and low-latency Internet services for terrestrial users globally . Network performance, which is typically affected by many practical factors such as the concrete implementation of network protocols and hardware capabilities, is very critical for satellite Internet service providers (SISP). Therefore, it should be important for SISPs to conduct laboratory characterization to benchmark and understand the network performance of their dedicated implementations of new network techniques before deploying them into the outer space. For example, a SISP may need to comprehensively and systematically assess the network performance of a new address allocation mechanism or a new routing policy in an experimental environment before the launch, and understand how well will these new techniques perform on existing SIC architecture in advance. Ideally, a laboratory benchmark environment (LBE) is expected to simultaneously accomplish fidelity and flexibility. However, existing benchmarking methodologies for terrestrial networks are insufficient to create a desired LBE for SICs due to several unique characteristics of SICs. First, due to the expensive manufacturing and launch cost, constructing an experimental satellite network using a number of real satellites should be technically and economically difficult. Second, benchmarking network performance of SICs via numerical or discrete-event-based simulation is fidelity-limited. Although network simulators can flexibly simulate satellite dynamics and constellation topology variation, they have limited capability to support the run of real system codes and network functions as in a real deployment. The abstraction-level of simulators might be too high to capture system-level effects as in real systems, such as power consumption and software overhead under heavy workloads. Finally, while network emulations can create virtual LBEs by integrating a number of virtual machines or containers to support the benchmark of real implementations of network protocols and functions, existing emulators are not constellation-consistent, because they inherently lack the ability of mimicking constellation-wide LEO dynamics and corresponding time-varying network behaviors as in a real SIC. This draft aims to provide basic considerations as specifications to guide network performance benchmark for SICs. Since an LEO satellite network constructed upon SICs has many unique characteristics as compared to existing terrestrial networks, our considerations in this draft include: (1) multiple networking models of emerging SICs; (2) a data-driven benchmarking approach that enables testers to build a LBE with acceptable flexibility and fidelity to support various test cases; (3) critical configuration parameters that might affect the SIC network performance; and (4) suggested test cases for SIC network performance benchmarking.

Notation and Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. This document uses the following acronyms and terminologies: SIC: Satellite Internet Constellation LEO: Low Earth Orbit SISP: Satellite Internet Service Provider LBE: Laboratory Benchmark Environment OSPF: Open Shortest Path First TCP: Transmission Control Protocol QUIC: Quick UDP Internet Connections SRLA: Satellite Relays for Last-mile Accessibility SRGS: Satellite Relays for Ground Station Networks GSSN: Ground Station Gateway for Satellite Networks DASN: Directly Accessed Satellite Networks GS: Ground Station SHF: Super High Frequency EHF: Extremely High Frequency GSaaS: Ground-Stations-as-a-Service VSAT: Very Small Aperture Terminal ISL: Inter-Satellite Link GSL: Ground-Satellite Link LoS: Line-of-Sight DUT: Device Under Test SUT: System Under Test

SIC Networking Models

SIC Components In particular, an emerging SIC typically includes a large number of low-flying broadband satellites, and geographically distributed ground facilities such as ground stations and user terminals (e.g. satellite dish). LEO broadband satellites relay and amplify radio telecommunication signals via transponders. These satellites can be equipped with high-speed radio and laser links , and thus promise to enable high-throughput inter-satellite and ground-satellite communication. To achieve low communication latency, emerging broadband satellites are operated in LEO to reduce the propagation latency. For example, the first phase of SpaceX's Starlink constellation is operated at about 550km altitude. As of September 2022, Starlink has already deployed more than 3000 mass-produced satellites with Ka-/Ku-/E-band phased array antennas and laser transponders (in some latest satellites). Ground stations are terrestrial radio stations designed for telecommunication with satellites. Typically, they are deployed on the earth surface, and communicate with satellites by transmitting and receiving radio telecommunication signals in the super high frequency (SHF) or extremely high frequency (EHF) bands. If a ground station successfully exchanges radio waves to an LEO satellite, it then establishes a telecommunication connectivity. Satellite Internet service providers often operate a large number of geo-distributed ground stations to control and coordinate their satellites. More recently, the world's leading cloud providers such as Amazon and Microsoft are actively deploying their Ground-Stations-as-a-Service (GSaaS) platforms , allowing satellite operators to use ground services on a flexible "pay-as-you-go" basis with affordable costs, and without the need to deploy their own ground infrastructures. User terminals, or very small aperture terminals (VSAT), satellite dishes, can be thought of as a special kind of small ground stations designed for connecting terrestrial users and satellites. In some practical SICs like the current form of Starlink, terrestrial users connect their handsets to broadband satellites via a signal conversion process performed by a dish-like terminal in the middle.

Networking Models of Emerging SICs At a high level, an LEO satellite network built upon SICs can be described as a dynamic graph, where each node presents a satellite, a ground station or a user terminal. A link connecting two ends in the graph refers to an inter-satellite link (ISL) or a ground-satellite link (GSL) in practice. The state of a link (i.e. active or inactive) might change over time, due to the dynamics of satellites and changes of inter-visibility. In practice, the concrete networking model, which describes how different components in an SIC are inter-connected to construct the network, could be different depending on the concrete SIC architecture and deployment. Based on the status quo of real-world commercial SICs and the latest academic literatures, we consider four representative SIC networking models for network performance benchmarking. (1) Satellite relays for last-mile accessibility (SRLA). Satellites and ground facilities can be integrated based on the classic "bent-pipe" architecture without the support of ISLs. In this model, satellites are used as relays to provide last-mile accessibility for terrestrial users. Specifically, user traffic from ground are first transmitted to the satellite, which then sends it right back down again like a bent pipe. This networking model is currently used by many ISIPs such as OneWeb. Figure 1 plots an example illustrating how two terrestrial users communicate with each other. During an end-to-end session, packets from the sender are first forwarded to a sender-side ground station, then to a receiver-side ground station through terrestrial Internet, and finally to the receiver by another satellite.

SRLA: satellite relays for last-mile accessibility. | Internet |<-->| Station | | Terminal| +---------+ +---------+ ------------- +---------+ +---------+ sender receiver ]]> (2) Satellite relays for ground station networks (SRGS) . Figure 2 depicts another "bent-pipe"-based inter-networking paradigm, where geo-distributed ground stations work as routers to construct a Layer-3 network. The only processing performed by satellites is to switch packets between two connected ground facilities. Note that in this networking model no satellites are equipped with ISLs. In a end-to-end communication session, packets from the sender is routed to the receiver by routes over satellites and ground stations.

SRGS: satellite relays for ground station networks. (3) Ground station gateway for satellite networks (GSSN) . Figure 3 shows another inter-networking approach based on ISLs. Leveraging ISLs, LEO satellites can build space routes to forward Internet traffic for long-haul communication, without the need of a large number of ground station relays. Ground stations work as an access point or a gateway for users. Satellites and ground stations jointly build a Layer-3 network for wide-area communication. During an end-to-end transmission, packets from the sender are first routed to a ground station via terrestrial networks, then to the receiver side ground station over satellite paths constructed by ISLs, and finally to the receiver by terrestrial network again. With inter-satellite communication enabled by ISLs, this networking model may require less ground stations as compared to SRLA and SRGS.

GSSN: ground station access for satellite networks. | Station | | Station |<----------->| Terminal| +---------+ Internet +---------+ +---------+ Internet +---------+ sender receiver ]]> (4) Directly accessed satellite networks (DASN) . Figure 4 plots another networking model where users install satellite terminals to directly access the satellite networks with ISL deployments, and can enable long-haul communication without the assistance of geo-distributed ground stations. In this model, satellite routers run dedicated space routing protocols to calculate their routing tables, and forward traffic from/to terrestrial users directly. Each satellite may also perform other network functions more than just routing, such as host configurations (e.g. IP, DNS allocation) for terrestrial user terminals.

DASN: satellite networks directly accessed by terrestrial users.

Considerations for SIC Benchmarking Methodology

LBE Requirements Ideally, a LBE built for benchmarking SIC network performance is expected to simultaneously accomplish acceptable realism, flexibility and cost. We summarize four baseline requirements as follows. (1) Constellation characteristics. The LBE is expected to mimic spatial and temporal constellation-wide characteristics of real mega-constellations. For example, the LBE is expected to be able to simulate/emulate network nodes at the same scale of a real mega-constellation, and can characterize the high dynamicity of LEO satellites, as well as its corresponding impact on network behaviors over time. (2) Network-level realism. The LBE is expected to support the run of real system codes and deploy the similar functionality like in a real system and networking stack. (3) Flexibility. As of the date of this writing, emerging mega-constellations are evolving rapidly, and many of them plan to launch hundreds to thousands more LEO satellites. Since a SISP's operating constellations might update frequently, the LBE is expected to flexibly support various network topologies at scale and load various network functions to meet various benchmarking requirements. (4) Usability. Finally, as we target at a laboratory-level benchmarking methodology, it is also expected that the LBE could be controllable, low-cost, and can provide easy-to-use programmable interfaces for testers to support diverse benchmarking requirements.

Exploiting A Data-driven Approach for SIC Benchmarking We consider a data-driven approach for creating a LBE that can satisfy the above requirements on benchmarking network performance of SICs. Our consideration is inspired by an important observation obtained from the current satellite Internet ecosystem: many organizations (e.g., regulators and satellite operators) and end users have shared a collection of public data to the community, including constellation regulatory information, orbital data observed from realistic satellites, ground station distributions and network capacities measured from terrestrial user terminals, etc. Based on this important fact, we consider to create a LBE for SIC benchmarking by judiciously combining real data trace, model-based orbit and network analysis, and large-scale network system emulation, to construct a real-data-driven digital twin, i.e., a virtual presentation synchronized to a real physical SIC in terrestrial environments for SIC benchmarking. In particular, the considered benchmarking approach can be summarized as follows. First, leveraging a crowd-sourcing approach to collect, combine and explore realistic constellation-relevant information to calculate the spatial and temporal characteristics consistent to real mega-constellations. Second, driven by such realistic information, exploiting a large number of networked virtual nodes and links to flexibly emulate a customized laboratory environment, which characterizes system-level effects and network behaviors consistent to a real SIC. Figure 5 depicts the overview of the considered data-driven approach for benchmarking network performance of SICs. The benchmarking environment consists four major components as follows.

A data-driven approach for benchmarking the network performance of SICs. | +----+----+----+----+----+ | |/SUT | traffic | | +-----+ | Satellite Network | | Emulator | +----------------------------+ ^ | +-----------------------+ | Traffic Generator | +-----------------------+ ]]> (1) A constellation-relevant information collector, which collects public constellation information and ground station distributions etc., from the satellite ecosystem. It maintains the key real-world information to support, guide and drive the construction of SIC benchmarking environments for various benchmarking requirements. (2) A satellite network emulator, which can calculate the spatial and temporal characteristics of a specific SIC, and further create a virtual SIC environment. It exploits VM- or container-based emulation to flexibly construct the virtual network environment based on concrete benchmarking requirements, and mimics satellite dynamics as well as the impact on network conditions (e.g. propagation latency change, connectivity loss and re-establishment). (3) A device under test (DUT) or system under test (SUT) which contains or runs the concrete implementation required for testing, and can connect to the virtual SIC environment to load interactive traffic. The DUT/SUT, together with the satellite network emulator, collaboratively construct the benchmarking environment. For example, in practice, the DUT/SUT can be a satellite hardware prototype running a tailored space routing mechanism required for testing. (4) A traffic generator that generates network traffic to drive the network performance benchmarking.

Benchmarking Workflow (1) Experiment preparation. A tester first prepares the concrete implementation for test, e.g. a new satellite routing program, or a new transport protocol implementation tailored for satellite Internet. (2) Benchmarking environment creation. Then the tester defines a network topology, i.e. a graph in which edges represent network links and nodes represent satellites, ground stations or end-hosts, and then create the SIC benchmarking environment. (3) DUT/SUT Deployment. Once the benchmark environment is constructed, in the deployment phase, the tester loads the implementation for testing on corresponding nodes in the environment. For example, if a tester needs to benchmark a new distributed routing program, then the routing implementation should be loaded on each emulated satellite in the virtual environment, and the DUT/SUT. Then the DUT/SUT is connected to the virtual environment. (4) Run test cases. Finally, run the dedicated test cases on the experimental network under specific application traffic. Performance results (e.g. latency, throughput, and route convergence time) can be measured for further in-depth analysis.

Benchmarking Scope The considered benchmarking approach mainly targets at benchmarking the network performance of a dedicated network technique as well as its system effects at various layers of the Internet protocol stack in an SIC. For example, evaluating a new routing/transport-layer protocol, or assessing the network performance of a new topology design in a highly-dynamic, resource constrained virtual SIC environment. The scale of the benchmark experiment supported by the considered approach is closely related to the underlying resources provided by underlying physical machines which are used to create the LBE.

Considerations for Benchmarking Environment Configuration Next we discuss the considerations for multiple configuration parameters of the benchmarking environment, which might be closely related to the benchmarking results.

Terminology and Definition of the Parameters

Parameters on Constellation Topology The topology of a constellation is jointly determined by many constellation-relevant parameters, including the orbit inclination, altitude, number of orbits, number of satellites in different orbits, connectivity pattern for inter-satellite and ground-satellite communication, number of ISLs in each satellite, etc. Inclination is the angle between an orbit and the Equator as the satellite moves. Typically, the value of inclination for polar orbits is about 90 degree. Altitude is a value measured over sea level and this value determines the orbital velocity of a satellite. Emerging SICs consist of low-flying satellites with altitude less than 2000km to enable low communication latency. The above orbital parameters, together with the number of orbits and the number of satellites, jointly affect the coverage of the satellite constellation. Connectivity pattern indicates how satellites should inter-connect to each other, and how satellites should connect to visible ground stations. There are two classic ISL connectivity patterns. +Grid suggests that each satellite connects to two adjacent satellites in the same orbit, and to other two satellites in adjacent orbits. Motif is a repetitive pattern where each satellite connects to multiple visible satellites and each satellite's local view is the same as that of any other.

Parameters on Ground Station Distribution There are three primary parameters related to ground stations, which might affect the benchmarking results. First, the geographical locations, which include latitude and longitude of ground stations. Second, the number of available antennas for space-ground communication. This value can affect the number of satellites that can be simultaneously connected by the ground station. Third, the minimum elevation angle, which determines the line-of-sight (LoS) of the ground station and can affect the available duration of space-ground communication.

Parameters on Network Links The total capacity of satellite communication systems has increased significantly over the past decade. Emerging broadband satellites can be equipped with high-speed radio or laser communication links. Link capacity is a critical parameter that can significantly affect the constellation-wide network performance of an SIC. Regarding the ground-to-satellite link capacity, during the beta test of Starlink, end users can achieve data speeds varying from 50Mbps (uplink) to 150Mbps (downlink) in most available locations. In addition, many planned constellations also suggest the use of laser inter-satellite links, which can potentially support up to tens or even hundreds of Gbps data transmission rate for inter-satellite communication . To reasonably benchmark the network performance of an SIC, a tester can configure the link capacity in the benchmark environment based on the concrete assessment requirements.

Setting of the Parameters We discuss different data-driven parameter settings based on best practices.

Constellation Orbital Parameters Two ways are used in practice, namely Regulatory-Data-Driven and Live-Data-Driven. Regulatory-Data-Driven Orbital Parameters SHOULD be tested and Live-Data-Driven Orbital Parameters are RECOMMENDED.

Regulatory-Data-Driven Orbital Parameters Orbital parameters of the constellations are reviewed and publicly disclosed by regulatory agencies (eg. FCC, ITU, etc.) and should be followed by the operators in principle, thus representing the ideal situation of the constellations. Both Polar-orbit and Inclined-orbit constellations SHOULD be tested. If the DUT/SUT is designed with orbital preferences, the preferences MUST be stated in the report. The table below provides the orbital parameters of the state-of-the-art networking constellations from regulatory agencies. Regulatory Data on Orbital Parameters of SoA Networking Constellations.

Name and Shell	Altitude (km)	Inclination (degree)	# of orbits	# of satellites per orbit	Polar / Inclined
Starlink S1	550	53	72	22	Inclined
Starlink S2	540	53.2	72	22	Inclined
Starlink S3	570	70	36	20	Inclined
Starlink S4	560	97.6	6	58	Polar
Starlink S5	560	97.6	4	43	Polar
Kuiper K1	630	51.9	34	34	Inclined
Kuiper K2	610	42	36	36	Inclined
Kuiper K3	590	33	28	28	Inclined
Telesat T1	1015	98.98	27	13	Polar
Telesat T2	1325	50.88	40	33	Inclined
OneWeb O1	1200	87.9	12	49	Polar
OneWeb O2	1200	55	8	16	Inclined

Live-Data-Driven Orbital Parameters Orbital Parameters can also be set based on live constellation GP data (general perturbations orbital data, also known for TLE) from CelesTrak.org . The GP data is produced by fitting observations (radar and optical) from US Space Surveillance Network (SSN) and provided continuously, thus representing the live situation of the constellations. Among GP and SupGP which are both provided, SupGP data is RECOMMENDED, as SupGP (Supplemental GP) is derived directly from owner/operator-supplied orbital data, providing reduced latency and improved accuracy comparing with GP. The Max Age of GP or SupGP SHALL be less than 1 day and MUST be less than 5 days. Comparing to Regulatory-Data, Live-Data is more accurate (in terms of per-satellite position), and also easy-to-get. However, Live-Data requires extra orbital determination process (implying inter-satellite relationship) to support network experiments. Once the orbital determination process is standardized, Live-Data-Driven Orbital Parameters shall SHOULD be used to benchmark.

Ground Station Distribution It's RECOMMENDED to set GS distribution based on Crowd-Sourcing-Data, which is often refined by fans community based on Regulatory-Data. For example, one crowd-sourcing global distribution of Starlink GSes could be found here , featuring details like the number of antennas and construction/opearation state of each GS. What's more, the data could be downloaded in KML format and feed into the banchmarking environment. Other OPTIONAL data for ground station distribution include Amazon AWS GS , Microsoft Azure Orbital GS , and SatNOGS , an open source global network of satellite ground-stations.

Connectivity Pattern Some of the connectivity patterns could be explored in live network and are RECOMMENDED to setup based on crowd-sourcing data. For other connectivity patterns, some RECOMMENDED strategies are also discussed in this section.

Crowd-Sourcing-Driven Connectivity Pattern It's RECOMMENDED to setup connectivity pattern based on crowd-sourcing data, if available crowd-sourcing data exists. For example, inter-ground station connectivity of Starlink ground stations is explored by the fans community , where the real users perform traceroute from all over the world and gather the results together. The data is also downloadable.

Strategy-based Connectivity Pattern For inter-satellite connectivity, "+Grid" strategy is widely-adopted and RECOMMENDED, where the satellites are connected with 4 neighbors and form a massive grid across the constellation. Other OPTIONAL inter-satellite connectivity strategies include "Inner-orbit Only" and "Motif" . For ground-to-satellite connectivity, "Nearest Ground Station with Antenna Quota" is intuitive and RECOMMENDED, Where each ground station is with 8 antenna quota is RECOMMENDED if there doesn't exist more specific data.

Network Link For more traditional network link setup, strategy-based setup is RECOMMENDED. For example, the propagation latency of ground-satellite link (RF) and inter-satellite link (free-space optical) could be derived from distance and light-speed. The capacity of ground-satellite link is RECOMMENDED to be set as 1 to 5 Gbps. Specific value MAY be derived from frequency band info from regulatory data. The capacity of inter-satellite link is RECOMMENDED to be set as 5 to 20 Gbps. Although measurement data on path latency and bandwidth from real satellite users are relative to network link setup, we didn’t find a good way to use directly. They may help on determining the coefficient when calculating link latency based on distance.

Considerations for SIC Test Cases In this section, we consider several test cases that can be used for benchmarking SIC network performance.

Benchmarking Routing Protocols in an SIC Network routing plays an important role in guaranteeing good service quality of SICs, since it not only determines the reachability between any two communication ends in the network, but also affects the achievable network performance perceived by customers. Ideally, an SIC routing mechanism is expected to simultaneously maintain high routing reachability for geo-distributed customers during the operation period, and provide low latency and high throughput paths for delivering various Internet traffic over the SIC. Therefore, it should be very important for satellite Internet service providers to benchmark how well will a routing protocol (and its implementation) perform in their SIC environment. Objective: given an implementation of the routing protocol for testing (e.g. OSPF , BGP or their variations optimized for space environments), this test case measures its network performance under a specific SIC configuration (e.g. the current form of the first phase of Starlink constellation which includes 1584 LEO satellites). Procedure: create an SIC network topology consisting of 1583 virtual satellites and a real DUT/SUT to emulate the satellite network. In addition, create two virtual user terminals in the virtual environment to emulate the source and destination in a communication session. Deploy the implementation for testing in each emulated satellite and the DUT/SUT. Run the tested routing implementation, and load traffic in the benchmarking environment to start the test. Measurement: since LEO satellites move in their orbits, the entire network topology should change over time. This test case measures the routing convergence time and the routing reachability under LEO dynamics.

Benchmarking Transport Protocols in an SIC Internet transport protocols, such as TCP and QUIC are expected to function correctly over any kinds of network paths. For satellite operators, it should be important to understand the network performance of transport protocols in an SIC network path. Note that the unique characteristics of SIC may impact network performance when using existing standard mechanisms. For example, in an SIC network, end-to-end latency might change due to the fluctuation of network paths caused by LEO high dynamics. Such a non-congestion latency increase might trigger cwnd shrinking for delay-based congestion control mechanisms such as TCP Reno. Objective: given an implementation of a transport protocol (e.g. TCP, QUIC or their variations optimized for satellite networks), measure its network performance under a specific SIC configuration. Procedure: create an SIC network topology consisting of 1583 virtual satellites and a real DUE device to emulate the satellite network. In addition, use the DUT/SUT as the source (e.g a TCP sender), and create one virtual user terminal in the virtual environment to emulate the destination (e.g. TCP receiver) in a communication session. Load traffic in the DUT/SUT to start the test. Measurement: This test case measures the performance of the tested transport protocol, such as end-to-end latency, jitter and throughput achieved in the transport layer.

Conclusion In this draft, we make several considerations as specifications for SIC network performance benchmarking. We describe multiple networking models of emerging SICs, a data-driven benchmarking approach which may enable testers to flexibly build a laboratory benchmark environment to support various test cases, critical configuration parameters that might affect the SIC network performance, and several suggested test cases for SIC benchmarking.

Acknowledgements

IANA Considerations This memo includes no request to IANA.

Security Considerations Benchmarking activities as described in this memo are limited to technology characterization using controlled devices in a laboratory environment, with dedicated address space and the constraints specified in the sections above. The benchmarking network topology as well as its parameter configurations will be an independent test setup, and the laboratory environment MUST NOT be connected to devices that may forward the test traffic into a production network, or misroute traffic to the test management network. In addition, benchmarking is performed on a "black-box" basis, relying solely on measurements observable external to the DUT/SUT. Special capabilities SHOULD NOT exist in the DUT/SUT specifically for benchmarking purposes. Any implications for network security arising from the DUT/SUT SHOULD be identical in the lab and in production networks.