Sequence ID calibration for
mis-ordered requests
Huawei Technologies Co.
The first operation is CREATE_SESSION. csa_sequence is one
argument of this operation, which is used for serializing
CREATE_SESSION via a per-client ID sequence number by the client. In
CREATE_SESSION request, csa_sequence should be equal to ether the
sequence ID in the client ID's slot(a retry), or the slot's sequence
ID + 1(correct normal request). Otherwise, NFS4ERR_SEQ_MISORDERED
will be returned from the server.
The second operation is SEQUENCE. sa_sequenceid is one argument
of this operation. In SEQUENCE request, If the difference between
sa_sequenceid and the server's cached sequence ID at the slot ID is
two (2) or more, or if sa_sequenceid is less than the cached
sequence ID (accounting for wraparound of the unsigned sequence ID
value), then the server <bcp14>MUST</bcp14> return
NFS4ERR_SEQ_MISORDERED.
The third operation is CB_ SEQUENCE, which is similar to
SEQUENCE. csa_sequenceid is one argument of this operation. In CB_
SEQUENCE request, If the difference between csa_sequenceid and the
client’s cached sequence ID at the slot ID is two (2) or more,
or if csa_sequenceid is less than the cached sequence ID (accounting
for wraparound of the unsigned sequence ID value), then the client
<bcp14>MUST</bcp14> return NFS4ERR_SEQ_MISORDERED.
Mis-order requests may happen as a result of network partition,
software bug, etc. For such request, the operations subsequent to
SEQUENCE, if any, are not processed, and so slots state (sequence ID,
cached reply) are not changed. That means, requests before this
mis-ordered one were processed correctly and the session state was
correct. In the current implementation, for most of the clients, this
error code will trigger the requester breaking the session and creating
a new session. This process unacceptably interferes with ongoing IO
operations, especially for the IOs on the normal slots.
For example of a persistent session, there are several slots on a
session. Requests on the slots are being processed correctly and replies
are being received normally. Suppose on one slot of them, a mis-ordered
request is received by the server and a response with
NFS4ERR_SEQ_MISORDERED error returns to the client. Then, the client is
going to destroy the session and establish a new session. Before the new
session is ready, new requests will not be performed until the pending
operations finished. The effects on IOs of normal slots will become
dramatic especially in network latency is substantial, as, for example
when client and server are in different locations. The client has to
break the session because it does not know what sequence is expected for
that session and slot. This current cached sequence information would be
available to the client eliminating any need to break the session.
Two operations, SEQENCE_QUERY and CB_SEQENCE_QUERY, are added to
query sequence ID cached when getting NFS4ERR_SEQ_MISORDERED error.
<CODE BEGINS>
/// /*
/// * structure for sequenceid query
/// */
/// struct SEQUENCE_QUERY4args {
/// sessionid4 sqa_sessionid;
/// slotid4 sqa_slotid;
/// };
<CODE ENDS>
<CODE BEGINS>
/// struct SEQUENCE_QUERY4resok {
/// sessionid4 sqr_sessionid;
/// slotid4 sqr_slotid;
/// sequenceid4 sqr_sequenceid;
/// };
/// union SEQUENCE_QUERY4res switch (nfsstat4 sqr_status) {
/// case NFS4_OK: SEQUENCE_QUERY4resok sqr_resok4;
/// default: void;
/// };
<CODE ENDS>
The SEQUENCE_QUERY operation is used by the client to query
sequence ID cached for designated session and slot.
SEQUENCE_QUERY <bcp14>MUST</bcp14> appear as the sole
operation type of any COMPOUND in which it appears. Multiple
SEQUENCE_QUERY operations can be used in one COMPOUND to query
multiple sequence IDs cached for multiple slots. The error
NFS4ERR_NOT_ONLY_OP will be returned when that constraint is not
met. Operations other than SEQUENCE, SEQENCE_QUERY,
BIND_CONN_TO_SESSION, EXCHANGE_ID, CREATE_SESSION, and
DESTROY_SESSION, <bcp14>MUST NOT</bcp14> appear as the
first operation in a COMPOUND.
If SEQUENCE_QUERY is received on a connection not associated with
the session via CREATE_SESSION or BIND_CONN_TO_SESSION, and
connection association enforcement is enabled (see Section 18.35),
then the server returns NFS4ERR_CONN_NOT_BOUND_TO_SESSION.
The sqa_sessionid argument identifies the session to which this
request applies. The sqr_sessionid result
<bcp14>MUST</bcp14> equal sqa_sessionid.
The sqa_slotid argument is the index in the reply cache for the
request. The sqr_slotid result <bcp14>MUST</bcp14> equal
sqa_slotid.
The sqr_sequenceid field is the cached sequence ID on the slot.
The client <bcp14>SHOULD</bcp14> use this value to
calibrate sa_sequenceid in the next SEQUENCE operation, that is,
sqr_sequenceid+1 <bcp14>SHOULD</bcp14> be used as the
sequence ID of the next request on this slot.
For CREATE_SESSION, SEQUENCE operations, if the sequence ID in
the request is mis-ordered(see RFC8881 18.46.3 Section), the replier
will fail the request by NFS4ERR_SEQ_MISORDERED and keep the reply
cache unchanged on the slot of this session. When getting
NFS4ERR_SEQ_MISORDERED error code in the response, the client
<bcp14>SHOULD</bcp14> query the cached sequence ID of
the slot and session by SEQUENCE_QUERY to calibrate its sequence ID
for the subsequent requests. That is, the sequence ID in next
request on this slot <bcp14>SHOULD</bcp14> be
sqr_sequenceid+1.
SEQUENCE_QUERY will leave the state of the slot (sequence ID,
cached reply) unchanged and lease of state related to the client ID
not renewed.
If the client is querying an unknown session ID to the server,
the server <bcp14>SHOULD</bcp14> return
NFS4ERR_BADSESSION in the response.
If the client is attempting to access a slot the replier does not
have in its slot table (It is possible the slot may have been
retired), NFS4ERR_BADSLOT <bcp14>SHOULD</bcp14> be
returned in the response.
<CODE BEGINS>
/// /*
/// * callback program structure for sequenceid query
/// */
/// struct CB_SEQUENCE_QUERY4args {
/// sessionid4 csqa_sessionid;
/// slotid4 csqa_slotid;
/// };
<CODE ENDS>
<CODE BEGINS>
/// struct CB_SEQUENCE_QUERY4resok {
/// sessionid4 csqr_sessionid;
/// slotid4 csqr_slotid;
/// sequenceid4 csqr_sequenceid;
/// };
/// union CB_SEQUENCE_QUERY4res switch (nfsstat4 csqr_status)
{
/// case NFS4_OK: CB_SEQUENCE_QUERY4resok csqr_resok4;
/// default: void;
/// };
<CODE ENDS>
CB_SEQUENCE_QUERY is used to calibrate sequence ID of the call
back request of the server for the backchannel of the session.
For each CB_COMPOUND request, the first operation
<bcp14>MUST</bcp14> be CB_SEQUENCE or CB_SEQUENCE_QUERY.
If any other operation is in the first position of CB_COMPOUND
except CB_SEQUENCE_QUERY and CB_SEQUENCE, NFS4ERR_OP_NOT_IN_SESSION
<bcp14>MUST</bcp14> be returned.
If the first operation is CB_SEQUENCE, CB_SEQUENCE
<bcp14>MUST</bcp14> appear once. The error
NFS4ERR_SEQUENCE_POS <bcp14>MUST</bcp14> be returned
when CB_SEQUENCE is found in any position in a CB_COMPOUND beyond
the first. If the first operation of a CB_COMPOUND is
CB_SEQUENCE_QUERY, CB_SEQUENCE_QUERY <bcp14>MUST</bcp14>
be the sole operation type. There can be multiple CB_SEQUENCE_QUERY
in this CB_COMPOUND to request multiple cached sequence IDs of
designated sessions and slots. If any other operations are found in
this CB_COMPOUND, NFS4ERR_NOT_ONLY_OP
<bcp14>MUST</bcp14> be returned.
The csqa_sessionid argument identifies the session to which this
request applies. The csqr_sessionid result
<bcp14>MUST</bcp14> equal csqa_sessionid.
The csqa_slotid argument is the index in the reply cache for the
request. The csqr_slotid result <bcp14>MUST</bcp14>
equal sqa_slotid.
The csqr_sequenceid field is the cached sequence ID on the slot.
The server <bcp14>SHOULD</bcp14> use this value to
calibrate csa_sequenceid in the next SEQUENCE operation, that is,
csqr_sequenceid+1 <bcp14>SHOULD</bcp14> be used as the
sequence ID of the next request on this slot.
For CB_SEQUENCE operations, if the sequence ID in the call back
request is mis-ordered(see RFC8881 20.9 Section), the client will
fail this request by NFS4ERR_SEQ_MISORDERED and keep the reply cache
unchanged on the slot of this session. When getting
NFS4ERR_SEQ_MISORDERED error code in the response, the server
<bcp14>SHOULD</bcp14> query the cached sequence ID of
the slot and session by CB_SEQUENCE_QUERY to calibrate its sequence
ID for the subsequent requests. That is, the sequence ID in next
call back request on this slot should be csqr_sequenceid+1.
CB_SEQUENCE_QUERY will leave the state of the slot (sequence ID,
cached reply) unchanged and the reply of CB_SEQUENCE_QUERY will not
renew the lease of state related to the client ID on the server
side.
If the server is querying an unknown session ID to the client,
the client <bcp14>SHOULD</bcp14> return
NFS4ERR_BADSESSION in the response.
If the server is attempting to access a slot the client does not
have in its slot table (It is possible the slot may have been
retired), NFS4ERR_BADSLOT <bcp14>SHOULD</bcp14> be
returned in the response.
This memo includes no request to IANA.
Note to RFC Editor: this section may be removed on publication as an
RFC.
All considerations from RFC 8881 security relative sections
[RFC8881].
The authors would like to acknowledge David Noveck for reviews of the
various versions of the draft.