]> Hammerspace

loghyr@gmail.com

General Network File System Version 4 Internet-Draft The Network File System version 4.2 (NFSv4.2) allows a client to cache data for file objects. Applications on some clients can control the caching of data, but there is no way to achieve this at a system level. This document introduces a new uncacheable file data attribute for NFSv4.2. Files marked as uncacheable file data SHOULD NOT have their data stored in client-side caches. This document extends NFSv4.2 (see RFC7862). Note to Readers Discussion of this draft takes place on the NFSv4 working group mailing list (nfsv4@ietf.org), which is archived at . Source code and issues list for this draft can be found at . Working Group information can be found at .

Introduction With a remote filesystem, the client typically caches file contents in order to improve performance. Several assumptions are made about the rate of change in the number of clients trying to concurrently access a file. With NFSv4.2, this could be mitigated by file delegations for the file contents. There are prior efforts to bypass file caching. In Highly Parallel Computing (HPC) workloads, file caching is bypassed in order to achieve consistent work flows and to allow concurrent access from many writers. Applications can use O_DIRECT on open (see ) to force the client to bypass the page cache, but the limitation is that each application must be modified to use this flag. This document introduces the uncacheable file data attribute to NFSv4.2 to bypass file caching on the client. As such, it is an OPTIONAL attribute to implement for NFSv4.2. However, if both the client and the server support this attribute, then the client SHOULD follow the semantics of the uncacheable file data attribute. The uncacheable file data attribute is read-write and per file. The data type is bool. A client can easily determine whether or not a server supports the uncacheable file data attribute with a simple GETATTR on any file. If the server does not support the uncacheable file data attribute, it will return an error of NFS4ERR_ATTRNOTSUPP. The only way that the server can determine that the client supports the attribute is if the client sends either a GETATTR or SETATTR with the uncacheable file data attribute. As bypassing file caching is file based, it is only applicable for objects which are of type attribute value of NF4REG. Using the process detailed in , the revisions in this document become an extension of NFSv4.2 . They are built on top of the external data representation (XDR) generated from .

Definitions

file caching

A client cache, normally called the page cache, which caches the contents of a regular file. Typical usage would be to accumulate changes to be bunched together for writing to the server.

write hole

A write hole is a data corruption scenario where either two clients are trying to write to the same erasure encoded block of dataor one client is overwriting an existing erasure encoded block of data. (Adapted from .) Note the hole occurs when multiple data servers are not consistent with the encoded block.

Further, the definitions of the following terms are referenced as follows:

• COMMIT (()
• file delegations ()
• GETATTR ()
• NF4REG ()
• NFS4ERR_ATTRNOTSUPP ()
• SETATTR ()
• system ()

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Caching of File Data The uncacheable file data attribute instructs the client to bypass its page cache for the file. This often includes write-behind caching used to combine multiple pending WRITEs into a single operation for more efficient remote processing. The uncacheable file data attribute, inhibits this behavior with an effect similar to the user using the O_DIRECT flag with the open call (). Under some conditions clients might be able to determine that files are not shared or do not exhibit access patterns suitable for write-behind caching. In such situations, setting this attribute provides a way to inform other clients of this judgment. The most pressing need for this feature is in connection with HPC workloads. These often involve massive data transfers and require extremely low latency. Write-behind caching can introduce unpredictable latency, as data is buffered and flushed later. Another aspect of such workloads is the need to share data between multiple writers. As the application data may span a data block in a page cache, data needs to be flushed immediately for the detection of write holes.

Uncacheable File Data If a file object is marked as uncacheable file data, all modifications to the file SHOULD be immediately sent from the client to the server. I.e., if a NFSv4.2 client fails to query this attribute, then it can not meet the requirements of the attribute. For uncacheable data, the client MUST NOT retain file data in its local data cache for the purpose of satisfying subsequent READ requests or delaying transmission of WRITE data. Reads MUST bypass the client data cache, and WRITE data MUST NOT be retained for read-after-write satisfaction or for the purpose of combining multiple WRITE requests. Caching of UNSTABLE WRITE data required to support the NFSv4.2 COMMIT operation is permitted and unaffected by this requirement. If the fattr4_uncacheable_file_data is not set when a client opens a file and is changed whilst the file is open, the client is not responsible for bypassing the page cache. It could flush the page cache and make all subsequent IO be direct, but until the client closes the file and reopens it, it is not required to meet the requirements of the attribute. If the client has a OPEN_DELEGATE_WRITE delegation on the file (see Section 10.4 of ), then the uncacheable file data attribute takes precedence over the caching of file data. The server can control this by not issuing file delegations for files with this attribute.

Setting the Uncacheable File Data Attribute The uncacheable file data attribute can allow for applications which do not support O_DIRECT to be able to use O_DIRECT semantics. One approach to support this would be to add a new parameter to mount that specifies all newly created files under that mount point would have their data not cacheable. Then the NFSv4.2 client would use a SETATTR to set fattr4_uncacheable_file_data. This approach is similar to the Solaris forcedirectio (See ) mount option.

Implementation Status There is a prototype Hammerspace server which implements the uncacheable file data attribute and a prototype Linux client which treats the uncacheable file data attribute as meaning use O_DIRECT. For the prototype, all files created under the mount point have the fattr4_uncacheable_file_data set to be true.

XDR for Uncacheable Attribute

Extraction of XDR This document contains the external data representation (XDR) description of the uncacheable file attribute. The XDR description is presented in a manner that facilitates easy extraction into a ready-to-compile format. To extract the machine-readable XDR description, use the following shell script: #!/bin/sh grep '^ *///' $* | sed 's?^ */// ??' | sed 's?^ *///$??' ]]> For example, if the script is named 'extract.sh' and this document is named 'spec.txt', execute the following command: sh extract.sh < spec.txt > uncacheable_prot.x ]]> This script removes leading blank spaces and the sentinel sequence '///' from each line. XDR descriptions with the sentinel sequence are embedded throughout the document. Note that the XDR code contained in this document depends on types from the NFSv4.2 nfs4_prot.x file (generated from ). This includes both nfs types that end with a 4, such as offset4, length4, etc., as well as more generic types such as uint32_t and uint64_t. While the XDR can be appended to that from , the code snippets should be placed in their appropriate sections within the existing XDR.

Security Considerations This document imposes no new security considerations to NFSv4.2.

IANA Considerations This document has no IANA actions.

References Normative References In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. This document describes the External Data Representation Standard (XDR) protocol as it is currently deployed and accepted. This document obsoletes RFC 1832. [STANDARDS-TRACK] This document describes NFS version 4 minor version 2; it describes the protocol extensions made from NFS version 4 minor version 1. Major extensions introduced in NFS version 4 minor version 2 include the following: Server-Side Copy, Application Input/Output (I/O) Advise, Space Reservations, Sparse Files, Application Data Blocks, and Labeled NFS. This document provides the External Data Representation (XDR) description for NFS version 4 minor version 2. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. This document describes the rules relating to the extension of the NFSv4 family of protocols. It covers the creation of minor versions, the addition of optional features to existing minor versions, and the correction of flaws in features already published as Proposed Standards. The rules relating to the construction of minor versions and the interaction of minor version implementations that appear in this document supersede the minor versioning rules in RFC 5661 and other RFCs defining minor versions. This document describes the Network File System (NFS) version 4 minor version 1, including features retained from the base protocol (NFS version 4 minor version 0, which is specified in RFC 7530) and protocol extensions made subsequently. The later minor version has no dependencies on NFS version 4 minor version 0, and is considered a separate protocol. This document obsoletes RFC 5661. It substantially revises the treatment of features relating to multi-server namespace, superseding the description of those features appearing in RFC 5661. Informative References Hammerspace Parallel NFS (pNFS) allows a separation between the metadata (onto a metadata server) and data (onto a storage device) for a file. The Flexible File Layout Type Version 2 is defined in this document as an extension to pNFS that allows the use of storage devices that require only a limited degree of interaction with the metadata server and use already-existing protocols. Data protection is also added to provide integrity. Both Client-side mirroring and the Erasure Coding algorithms are used for data protection. Linux man-pages project Linux man-pages project Linux man-pages project Oracle Solaris Documentation

Acknowledgments Trond Myklebust, Mike Snitzer, and Thomas Haynes all worked on the prototype at Hammerspace. Rick Macklem, Chuck Lever, and Dave Noveck reviewed the document. Chris Inacio, Brian Pawlowski, and Gorry Fairhurst helped guide this process.