]> Viagenie

Canada marc.blanchet@viagenie.ca

Aalyria Technologies

USA wes@aalyria.com

Juniper Networks

USA tony.li@tony.li

Internet Internet Engineering Task Force dtn quic space deep space ip dns routing Deep space communications involve long delays (e.g., Earth to Mars is 4-24 minutes) and intermittent communications, because of orbital dynamics. The IP protocol stack used on Earth's Internet is based on assumptions of shorter delays and mostly uninterrupted communications. This document describes the architecture of the IP protocol stack tailored for its use in deep space. It involves buffering IP packets in IP forwarders facing intermittent links and adjusting transport protocol configuration and application protocol timers. This architecture applies to the Moon, Mars, or general interplanetary networking.

Introduction Deep space communications involve long delays (e.g., Earth to Mars is 4-20 minutes) and intermittent communications, because of orbital dynamics. Up to now, communications have been done on a layer-2 point-to-point basis, with sometimes the use of relays, but no layer-3 networking has been in routine use. reports an assessment done around 25 years ago concluding that the IP protocol stack was not suitable for deep space networking. However, some things have changed since that time, as there has been evolution in Internet transport and security protocols (e.g. QUIC), routing (e.g. SDN), and forwarding technology (e.g. MPLS and Segment Routing). Currently, space agencies have published plans that include deploying IP networks on celestial bodies, such as the Moon or Mars, on the surface and in orbital vicinity, including layer 2 technologies such as Wi-Fi or 5G. On the surface, the plans involve dense networking around facilities and habitats. New mission concepts are also including clusters of multiple networked nodes co-located at Lagrange points. A previous document revisited the initial assessment of not using IP and concluded that the IP stack is viable in deep space, given the IP stack evolution that happened since the original evaluation. This document defines an architecture to use IP in deep space networking. IP in deep space means running IP over deep space layer-2 links, a reliable transport over IP, applications protocols over that transport, and applying proper routing, security, and network management on that IP network. Reusing the whole IP stack in deep space enables the reuse of all protocols, tools, and software currently used on the Internet. However, as one might already argue, many components of the IP stack cannot be used as is and therefore requires careful configuration and deployment considerations that are discussed in this document. Since the Moon is a few light seconds away from Earth, it is possible to configure and run various IP-based protocols and applications to make it "work". Mars with a much longer delay is more difficult. This framework would also work for longer delays, such as reaching Jupiter or the whole Solar System Internet (SSI), but it is not specifically discussed. This document uses "deep space" extensively as opposed to "space" which often includes earth-orbiting communications, which are not covered in this document. Even if the definition of deep space per the ITU does not include the Moon, this document applies to IP networks on the Moon. This framework proposes to use IP in deep space, based on queueing outbound packets for upcoming contact opportunities. In this case, the IP layer has to deal with the fact that a next-hop may not be currently reachable and that IP packets could be buffered for an unusual amount of time, such as minutes, hours, or days, in the forwarding device waiting for reachability back because of a new link-up opportunity. The transport layer should be able to work with long and variable delays, including intermittent communications. The application protocols and the applications themselves should be properly set to wait a longer time than on the current Internet to receive a response to a query. Finally, all network services such as routing, security, naming, and network management should also be adapted to this new context. This document is structured around these layers. The key characteristics of space communications and networking, its use case and its requirements are discussed in another document.

Document and Discussion Location The source of this document is located at https://github.com/marcblanchet/draft-deepspace-ip-architecture. Comments or changes are welcomed using a PR or an issue. This subject should be discussed on the deepspace@ietf.org mailing list.

Layer 2 in Deep Space

Celestial Body Surface The Interagency Operations Advisory Group (IOAG) has defined the communications architecture for the Moon and Mars. On the celestial body surface, it is planned to use 3GPP and Wi-Fi link layer protocols. IP will be used over these link layers.

Deep Space Links Deep space links typically use the Consultative Committee for Space Data Standards (CCSDS) standards for link layers, such as Telecommand (TC) , Telemetry (TM) , Advanced Orbiting Systems (AOS) , Proximity1 (Prox1) or the Unified Space Data Link Protocol (USLP) . CCSDS has defined a generic encapsulation mechanism for the payloads for all these link layer protocols with IP as an encapsulated protocol . Therefore, IP packets can be transported over any CCSDS link layers.

Celestial Body Orbits For celestial body orbits, IOAG has planned the use of CCSDS link layer protocols. However, as on Earth, it may be possible to use 6G-NTN technology around celestial bodies, such as in lunar or Martian orbits. 6G-NTN technologies use IP as its layer 3 technology.

Internet Protocol IPv4 or IPv6 packets can be carried as is over long delays and disruptions, as IP has no notion of time. Originally, the Time To Live (TTL) field of IPv4 was defined based on time , but it has been effectively implemented as a hop count, which was renamed as "Hop Count" in IPv6 . Nothing needs to be changed to the IP protocol or its packet format.

IP Forwarding and Store-and-Forward For deep space applications, an IP packet may need to be queued for much longer periods than are typical for the Internet when the next hop is currently unreachable or undefined, which can happen due to planning of periodic contacts around orbital dynamics, and other antenna scheduling limitations. Terms have been used including "store-and-forward" (which is already used differently elsewhere in the context of switch architectures), or "store, carry, and forward", to describe the behavior of buffering IP packets rather than immediately discarding them when the next-hop is not immediately available on-link. This queuing may be implemented at layer 2 as is currently done by the Mars orbiters. In this case, the frames are stored, regardless of payload type. In this case, IP packets are unaware of the store-and-forward and no changes are needed in the IP forwarding function. The L2 network is just behaving as a point-to-point link with a large and variable latency. If an IP forwarder has an interface on an intermittent link, and that link is down, some destinations may become unreachable when there is no alternate route. In this case, the forwarder buffers the IP packets locally instead of dropping them. This might be implemented as a deep queue with active queue management (AQM) . When the route to the destination is back, on the same link or a different link, maybe minutes or hours later, the stored packets can be transmitted. This requires proper provisioning of buffer storage memory for the target deployment and usage. If the buffer is full, then packets must be dropped. The choice of which packets to drop depends on the policies configured on the node, which may be a function of traffic class, source or destination IP addresses, flow labels, or other parameters. An example is described in . Packets might also be lost if a buffer is cleared for any other reason (e.g. due to reboot), or corrupted somehow (e.g. due to cosmic rays or other uncorrectable memory upsets). The architecture described in this document does not require IP forwarding nodes to necessarily implement anything beyond a typical "best effort" service and reliability, though more complex means to support reliable packet delivery are compatible and can interoperate within the architecture if desirable.

Header Compression Deep space links are point-to-point links and bandwidth in space is very valuable, so header compression is very effective. Static Context Header Compression (SCHC) is a header compression technique that relies on rules in a static context and is, therefore, more efficient for deep space. SCHC should be considered on a deep space point-to-point link or a string of L2 links.

IP Addressing and Routing

The IP address space is a hierarchical namespace where ranges of addresses are encoded as "prefixes". Individual domains advertise prefixes to the broader Internet and assign these addresses internally. Prefixes may be aggregated into less-specific prefixes, which makes the routing subsystem more efficient by decreasing overhead. Space networks provide a unique opportunity to provide extremely efficient routing by assigning a unique prefix or block of addresses per celestial body and its proximal orbits. Management of the IP address space is currently documented in , but this only covers continental regions and does not provide for addressing for space. Address space for outer space should be managed by a Regional Internet Registry (RIR) and blocks of address space should be allocated for each celestial body of interest. Space service providers should use prefixes assigned by this RIR.

Existing routing protocols require proof of liveness between protocol partners, implemented through the periodic exchange of packets between partners. This is impractical on long-delay or intermittent links, so a PCE based approach seems appropriate for those domains possibly supplemented by contact plan schedules. Interconnection between domains can still be done with BGP , but long-delay or intermittent links should be avoided. Domains straddling such links must provide proxy advertisements for prefixes reachable across such links. Optimal routing for domains with intermittent links is out of scope for this document. On the surface of celestial bodies and in proximal orbit, traditional protocols are applicable and should be used (e.g., ).

Transport Modern transport protocols and stacks share similar algorithms for common transport needs. This section first addresses general transport issues, and then addresses use of specific individual transport protocols. While there have been academic research experiments in developing new protocols specifically for interplanetary transport, they are not directly considered here because the goal is to leverage IETF protocols and commonly available software.

General Transport Issues Internet transport protocols and transport stacks have developed to meet the needs for different classes of applications on the terrestrial Internet, using similar algorithms with parameters that are tuned for typical conditions. Some of these algorithms might simply be parameterized differently for interplanetary network paths, while other algorithms may have fundamental challenges. In some cases, minor adjustments (or no adjustment) will suffice for Earth-lunar networking.

• Protocol Negotiation / "Happy Eyeballs" - Due to presence of both IPv4 and IPv6 in the terrestrial Internet, applications or transport stacks may include "happy eyeballs" capabilities that make use of multiple DNS queries and connection attempts for different addresses and address families returned. Downsides to this in interplanetary applications might include (1) the additional load on interplanetary DNS (although solutions to this are avialable ), (2) unnecessary use of network capacity for paralel connection attempts (e.g. if 0-RTT data transmission is used), and (3) unnecessary additional transport state maintained for an extended period of time. If applications for interplanetary use initially rely on either fixed IP addresses (instead of DNS names) or a single address family (e.g. IPv6), then there should be no downsides to presence of happy eyeballs algorithms within the transport stack. If happy eyeballs is present and triggered, then the values in section 8 of should be carefully considered and tuned based on the DNS and path expectations (e.g. resolution delay, first address family count, connection attmept delay, minimum connection attempt delay, and maximum connection attempt delay) as most of these values are not likely to be appropriate even for Earth-lunar paths.
• Connection Initiation / Handshaking - Very long-lived connection state may be used to avoid connection initiation in some cases, e.g. by establishing state prior to launch and using those pre-established connections long-term. However, this will not be possible in all cases for all applications, if flows can't be planned pre-launch. Due to the need to be robust to stale packets, errors, and denial of service attacks, historically, Internet transports have included handshaking state machines, such as 3-way and 4-way handshakes for connection establishment (the case can be even worse for some transport protocol and TLS combinations), although QUIC can established secured connections with only 1 round-trip time. Since the interplanetary round trip times may be larger than the duration of contact periods, these handshaking mechanisms are very inefficient. Though they are tolerable in cases such as between Earth and lunar networks, they are stifling for Earth-Mars and other interplanetary network paths. Transports, such as QUIC, that have the ability to resume based on shared state from prior application connections or to rapidly start transferring data ("0RTT") can be suitably efficient, once initial state is obtained; however, they still require a complete initial handshake with the full set of round trip times imposed. For interplanetary use, it may be beneficial to find ways to securely pre-set information to allow this more efficient startup, without requiring the full initial handshake even once.
• Capability / Feature Negotiation - Ideally, transport protocol feature advertisements and agreements can be completed prior to launch as part of connection pre-establishment and cached long-term. Any negotiation of additional features that requires full round trip times is prohibitive for general interplanetary use, especially if data transmission is stalled while negotiation takes place. Commonly, much negotiation can occur in the initial connection handshake. It will be beneficial if transport stacks can cache (or securely pre-configure caches) of pre-negotiated features with typical hosts that they plan to communicate with during the course of a mission.
• Retransmission - Reliable transport protocols typically keep retransmission timers, computed based on round trip time samples . Since it may be impossible to even take RTT measurements within a contact window, and RTTs may vary significantly across contact windows, this type of algorithm is not appropriate for interplanetar use. Additionally, default values may be in ranges such as 1 second, that are inappropriate for even Earth-lunar paths. Based on the known propagation times for interplanetary paths and predicted contacts, either pre-configured values or new algorithms should be used.
• Handling Failures - Aside from retransmission, there are also other types of timers and counters in transport stacks, for things including DNS resolution, connection establishment retries, connection keep-alives, user timeouts, and delayed acknowledgement. Additionally, transports receive and respond to ICMP messages that indicate other different types of errors from within the network. Similar to the case of retransmission timers, other types of timers that support failure handling should be adjusted based on estimated propagation and forwarding times. Other heuristics for validating ICMP messages and responding may need to be considered.
• Congestion Control - Internet congestion control algorithms are based on timely receipt of feedback in order to detect losses, delays, or other signals, and typically attempt to react rapidly. Two challenges in the interplanetary case include: (1) The relative delays on interplanetary paths are much longer (preventing "rapid" response) and may vary in orders of magnitude between segments, e.g. across a terrestrial segment, an Earth-Mars segment, and a Mars orbit/surface segment. Congestion may occur in low-latency portions of the network, but fail to be detected quickly because loss/acknowledgement information propagates and feeds back too slowly, leading to sustained loss due to unresponsiveness in a low-latency portion of the network. (2) By the time feedback is received, it is badly out of date, if not completely irrelevant to the future. The advent of IP-based in-network storage for scheduled interplanetary links also changes the way that congestion can be measured (e.g. in delay-based protocols). In the near-term, and in present space mission operations, it can be assumed that data link capacity is explicitly planned and scheduled end-to-end in order to accomodate mission needs. This makes congestion control algorithms largely replaceable with simple nominal rate and flow control. However, for larger and more dynamic future interplanetary networks, and shared trunk links, etc., it will be useful in the future to develop more optimized congestion control methods, including possibly more effective network-based signaling/feedback, rather than simply end-to-end feedback-based mechanisms.
• Path MTU Discovery - Internet transports typically include probing algorithms and rely on end-to-end acknowledgements (or in legacy cases ICMP errors) in order to infer the maximum packet sizes that can be used at any time without introducing unnecessary IP fragmentation. Because for interplanetary cases, feedback may be slow or unrelated to the current or future paths by the time it arrives, it may be more beneficial to simply rely on known / pre-arranged path MTU limitations that can be communicated between interplanetary providers and other connected providers and users. Especially for higher bandwidth links, it may be beneficial for this to be significantly larger than the minimum Internet MTU values that are normally assumed. For instance, terrestrially many systems use 9000+ byte MTUs. MTUs in interplanetary networks may vary due to the range of link bandwidths, and simultaneous desires to be efficient on fast links while also wanting to avoid head-of-line blocking for large packets on low-bandwidth links.
• Multiple Streams - Several transports allow for applications to send/receive multiple independent streams of application data, rather than requiring separate connections (and handshakes, and state) for each stream. Due to the need to leverage pre-established state and make efficient use of connectivity opportunities, these capabilities can be valuable to leverage for interplanetary use, however, specific mechanisms may need to be evaluated/tuned/modified e.g. to avoid additional round trip times.
• Multipath Transport - There may be a variety of different physical paths between planets (e.g. using relays or direct links), that could be attractive for a mission to use simultaneously as way to increase reliability for mission data, or to increase throughput for an application. However, typical Internet multipath transport algorithms are oriented more towards failover than towards replicating or load-balancing traffic. They may only switch flows between using alternative paths (e.g. for failover), or allow different streams to use different paths, rather than replicating data. For deep space use, Internet multipath transport algorithms could either be tuned or replaced.
• Forward Error Correction (FEC) - Generally, FEC is expected to be implemented below the link layer. Within the transport layer, FEC and/or erasure coding are not common in typical Internet use today, though FEC and/or erasure coding can be integrated with different protocol stacks. Because it can allow for data recovery in the case of packet losses, without suffering extra round trips, or requiring bidirectional connectivity, this may be valuable to incorporate/enable in future interplanetary transport stacks.

UDP UDP has no notion of time and, therefore can be used as-is in deep space. Hence, protocols using UDP transport can be used in space as-is, if they do not rely on time or can be configured with timeouts appropriate in deep space.

QUIC QUIC like most IP transport protocols implements congestion control mechanisms, which, based on various metrics such as calculated delays or packet loss, pace the rate of sending packets at the source node to decrease the perceived congestion in the network. QUIC supports many new features suitable and useful in deep space such as 1 RTT for connection establishment and security, mobility, 0 RTT, streams, user space, etc. [TLI: This sentence needs more words to explain these references.] Current implementations of QUIC typically set various transport configuration parameters suitable for the Internet environment, expecting an RTT to be in the hundreds of milliseconds and a normally always-connected network. Therefore, QUIC stacks using default configurations will not work in deep space. However, studies and simulations showed that with proper configuration of parameters, QUIC stacks can support the delays and disruptions in deep space. describes how to properly configure a QUIC stack for deep space applications, where QUIC is unaware of disruptions. If the transport is aware of the disruptions, further optimizations are possible. Having multiple streams and applications within a single QUIC connection is valuable and useful for deep space. A ground station may set up the initial QUIC connection with a spacecraft and then carry all needed applications and streams over that same connection for the whole duration of the mission. Session keys and certificate lifetimes together with certificate validation and trust chain anchors need to be carefully configured and handled. QUIC proxies can be used at the edge of space to isolate, apply policies, or optimize traffic at the ingress/egress to a celestial body network.

Other Transports Other transports such as TCP , SCTP , DCCP and others were not investigated for their suitability in space.

HTTP HTTP by itself has no notion of time. An HTTP request and response may take minutes or hours to be completed. However, current infrastructure and software on the Internet have various time-related configurations that will not work well in the deep space context. HTTP headers containing time, such as Cache-Control and Expires , should not be used or if used, must be set to large enough values to cover the longest delay so that expiration does not happen before the actual data arrives at the destination. As with any HTTP application and content on the Internet, these headers should be set properly based on the deployment use case, which is even more important for deep space. Similarly, when continuous content transfer is used, as with 100-Continue , proper values for headers should be set. HTTP clients and servers typically have default timeouts that should be modified. For example, curl has the "-m" option for this use case. Similarly, HTTP server implementations have various timeout configuration variables that must be set properly. Testing with HTTP client Curl and HTTP server nginx and an introduced network delay of minutes, hours and days showed that HTTP communications work well with basic configuration changes. HTTP applications themselves must be developed using an asynchronous pattern and if they have timeouts, they should be adjusted appropriately. Internet websites are designed with the assumption of hundreds of milliseconds delay and relatively always connected, where pages contain multiple queries to get further resources, media, queries to web services, and downloading additional code and frameworks. This could work in theory in space, but it will not be optimal, as multiple queries will be generated and therefore take multiple RTT before the whole page is received complete. This issue can be mitigated by using various techniques such as Web Assembly or pre-caching. Moreover, it could be possible to have simple HTML pages with no or very few references and no media content that was not locally cached. An example would be a rover on Mars presenting an HTTP server with a base and bare HTML page to offer basic info on its status (maybe all in text) and some additional detailed pages, most likely also in base HTML text. However, it is foreseen that most applications based on QUIC-HTTP transport in deep space would be using REST or similar asynchronous patterns and not typical web browsing. Caching should be used extensively on space networks to maximize local fetching. Preemptive caching by pre-populating caches with data that shall be used locally on the celestial body network shall be done as much as possible to provide better response time on the local celestial body network. QPACK should be considered for higher bandwidth efficiency.

CoAP The Constrained Application Protocol (CoAP) is a specialized web transfer protocol for use with constrained nodes and constrained networks, therefore a candidate for an application protocol for space, similar to HTTP. If considered, proper use of its underlying transport and timers should be looked at.

Network services

Naming For small-scale deployments, one can use IP addresses directly or a mapping from a name to an IP address such as /etc/hosts. However, this does not provide easy dynamic updates, scaling by hierarchy, service discovery, authentication of records, etc. Therefore, the Domain Name System (DNS) shall be considered early on in the space deployment. However, naming hierarchy and infrastructure must be carefully designed to avoid name resolution over deep space links, given that answers may come after minutes or hours. There are clear advantages of having the space name hierarchy anchored to the current Internet root, as it enables DNSSEC through the same security infrastructure currently used and deployed. Using the same root also does not require new policies. A new TLD or a new root is way more complicated and does not bring any significant value compared to using the current domain tree. Care must be taken to manage key lifecycles and resource record lifetimes. discusses the various methods and the naming hierarchy that should be used in space.

Network Management NETCONF and RESTCONF shall be used with proper configuration values to avoid timeouts and appropriate transport. NETCONF over QUIC transport or RESTCONF over HTTP over QUIC transport shall be configured with appropriate QUIC transport parameters as discussed in . Given the non-typical delays in requests and response in space compared to traditional network management frameworks on Internet and private networks, expectations about when configuration changes will be applied and confirmed or the timeliness of the actual value received from a request need to be taken into account. For example, a configuration change may take hours to be received by the spacecraft, which is not expected in typical network operations. A request for some value may take hours to be received by the spacecraft and take hours to be received by the requestor, which means the value may not be current or expired. Moreover, typical timeouts of NETCONF clients should be adjusted to the expected RTT. While being declared historic in IETF, SNMP runs over UDP and has no notion of time. Therefore, with proper configuration of client timeout, it can be used as is to manage nodes and services in deep space.

IANA Considerations This memo includes no request to IANA.

Security Considerations Using the current IP protocol stack in deep space inherits all the work on privacy, cryptography, key management, firewalls, and scrutiny of protocols that are deployed on the Internet. As an example, TLS has been more carefully examined than almost any other secure transport protocol. Moreover, given that no changes are made in the protocols, this architecture does not bring new security issues on the protocols themselves. Deep space security requirements are different than on the existing Internet, but nothing has been found to prevent the conformance of the IP protocol stack to those requirements. As it is currently planned, the deep space network shall be isolated from the current Internet by an "air gap", to disable any direct communications from the Internet to deep space. Moreover, destination IP prefix filtering shall be used to restrict the traffic to only the relevant one for each link. Note that this shall also be implemented in the routing control plane, but additional security might be appropriate to further protect the deep space links. Each celestial network edge device shall have firewall rules to prevent inappropriate traffic from entering deep space links. If communications from Mars may only occur to Earth, but not to the Moon, then appropriate filtering based on destination IP prefixes shall be used. Storage in IP forwarders may become full by normal traffic or by malicious traffic that could become a denial-of-service attack. Appropriate policies and measures should be put in place in those forwarders to drop packets in advance to avoid the depletion of storage space and to mitigate such attacks.

References Informative References Consultative Committee on Space Data Systems(CCSDS) Consultative Committee on Space Data Systems(CCSDS) Consultative Committee on Space Data Systems(CCSDS) Consultative Committee on Space Data Systems(CCSDS) Consultative Committee on Space Data Systems(CCSDS) Consultative Committee on Space Data Systems(CCSDS) World Wide Web Consortium(W3C) Lunar Communications Architecture Working Group, Interagency Operations Advisory Group Mars and Beyond Communications Architecture Working Group, Interagency Operations Advisory Group CCSDS Viagenie

Acknowledgements This work started by reassessing the use of the whole IP stack in the context of deep space discussed in where early contributors are acknowledged. This document and its underlying work has been reviewed and discussed by many, who have provided valuable feedback and comments, including disagreements, and made an overall more solid document. These people are, in no specific order: Padme Pillay-Esnault, Marius Feldmann, Britta Hale.