xxHash fast digest algorithm

Introduction This document describes the xxHash digest algorithm for both 32-bit and 64-bit variants, named XXH32 and XXH64. The algorithm takes an input a message of arbitrary length and an optional seed value, then produces an output of 32 or 64-bit as "fingerprint" or "digest". xxHash is primarily designed for speed. It is labeled non-cryptographic, and is not meant to avoid intentional collisions (same digest for 2 different messages), or to prevent producing a message with a predefined digest. XXH32 is designed to be fast on 32-bit machines. XXH64 is designed to be fast on 64-bit machines. Both variants produce different output. However, a given variant shall produce exactly the same output, irrespective of the cpu / os used. In particular, the result remains identical whatever the endianness and width of the cpu is.

Operation notations All operations are performed modulo {32,64} bits. Arithmetic overflows are expected. XXH32 uses 32-bit modular operations. XXH64 and XXH3 use 64-bit modular operations. When an operation ingests input or secret as multi-bytes values, it reads it using little-endian convention. +: denotes modular addition -: denotes modular subtraction *: denotes modular multiplication Exception: In XXH3, if it is in the form (u128)x * (u128)y, it denotes 64-bit by 64-bit normal multiplication into a full 128-bit result. X <<< s: denotes the value obtained by circularly shifting (rotating) X left by s bit positions. X >> s: denotes the value obtained by shifting X right by s bit positions. Upper s bits become 0. X << s: denotes the value obtained by shifting X left by s bit positions. Lower s bits become 0. X xor Y: denotes the bit-wise XOR of X and Y (same width). X | Y: denotes the bit-wise OR of X and Y (same width). ~X: denotes the bit-wise negation of X.

XXH32 Algorithm Description

Overview We begin by supposing that we have a message of any length L as input, and that we wish to find its digest. Here L is an arbitrary nonnegative integer; L may be zero. The following steps are performed to compute the digest of the message. The algorithm collect and transform input in stripes of 16 bytes. The transforms are stored inside 4 "accumulators", each one storing an unsigned 32-bit value. Each accumulator can be processed independently in parallel, speeding up processing for cpu with multiple execution units. The algorithm uses 32-bits addition, multiplication, rotate, shift and xor operations. Many operations require some 32-bits prime number constants, all defined below:

These constants are prime numbers, and feature a good mix of bits 1 and 0, neither too regular, nor too dissymmetric. These properties help dispersion capabilities.

Step 1. Initialize internal accumulators Each accumulator gets an initial value based on optional seed input. Since the seed is optional, it can be 0.

Special case: input is less than 16 bytes When the input is too small (< 16 bytes), the algorithm will not process any stripes. Consequently, it will not make use of parallel accumulators. In this case, a simplified initialization is performed, using a single accumulator:

The algorithm then proceeds directly to step 4.

Step 2. Process stripes A stripe is a contiguous segment of 16 bytes. It is evenly divided into 4 lanes, of 4 bytes each. The first lane is used to update accumulator 1, the second lane is used to update accumulator 2, and so on. Each lane read its associated 32-bit value using little-endian convention. For each {lane, accumulator}, the update process is called a round, and applies the following formula:

This shuffles the bits so that any bit from input lane impacts several bits in output accumulator. All operations are performed modulo 2^32. Input is consumed one full stripe at a time. Step 2 is looped as many times as necessary to consume the whole input, except for the last remaining bytes which cannot form a stripe (< 16 bytes). When that happens, move to step 3.

Step 3. Accumulator convergence All 4 lane accumulators from the previous steps are merged to produce a single remaining accumulator of the same width (32-bit). The associated formula is as follows:

Step 4. Add input length The input total length is presumed known at this stage. This step is just about adding the length to accumulator, so that it participates to final mixing.

Note that, if input length is so large that it requires more than 32-bits, only the lower 32-bits are added to the accumulator.

Step 5. Consume remaining input There may be up to 15 bytes remaining to consume from the input. The final stage will digest them according to following pseudo-code:

= 4) { lane = read_32bit_little_endian(input_ptr); acc = acc + lane * PRIME32_3; acc = (acc <<< 17) * PRIME32_4; input_ptr += 4; remainingLength -= 4; } while (remainingLength >= 1) { lane = read_byte(input_ptr); acc = acc + lane * PRIME32_5; acc = (acc <<< 11) * PRIME32_1; input_ptr += 1; remainingLength -= 1; } ]]> This process ensures that all input bytes are present in the final mix.

Step 6. Final mix (avalanche) The final mix ensures that all input bits have a chance to impact any bit in the output digest, resulting in an unbiased distribution. This is also called avalanche effect.

> 15); acc = acc * PRIME32_2; acc = acc xor (acc >> 13); acc = acc * PRIME32_3; acc = acc xor (acc >> 16); ]]>

Step 7. Output The XXH32() function produces an unsigned 32-bit value as output. For systems which require to store and/or display the result in binary or hexadecimal format, the canonical format is defined to reproduce the same value as the natural decimal format, hence follows big-endian convention (most significant byte first).

XXH64 Algorithm Description

Overview XXH64's algorithm structure is very similar to XXH32 one. The major difference is that XXH64 uses 64-bit arithmetic, speeding up memory transfer for 64-bit compliant systems, but also relying on cpu capability to efficiently perform 64-bit operations. The algorithm collects and transforms input in stripes of 32 bytes. The transforms are stored inside 4 "accumulators", each one storing an unsigned 64-bit value. Each accumulator can be processed independently in parallel, speeding up processing for cpu with multiple execution units. The algorithm uses 64-bit addition, multiplication, rotate, shift and xor operations. Many operations require some 64-bit prime number constants, all defined below:

These constants are prime numbers, and feature a good mix of bits 1 and 0, neither too regular, nor too dissymmetric. These properties help dispersion capabilities.

Step 1. Initialize internal accumulators Each accumulator gets an initial value based on optional seed input. Since the seed is optional, it can be 0.

Special case: input is less than 32 bytes When the input is too small (< 32 bytes), the algorithm will not process any stripes. Consequently, it will not make use of parallel accumulators. In this case, a simplified initialization is performed, using a single accumulator:

The algorithm then proceeds directly to step 4.

Step 2. Process stripes A stripe is a contiguous segment of 32 bytes. It is evenly divided into 4 lanes, of 8 bytes each. The first lane is used to update accumulator 1, the second lane is used to update accumulator 2, and so on. Each lane read its associated 64-bit value using little-endian convention. For each {lane, accumulator}, the update process is called a round, and applies the following formula:

This shuffles the bits so that any bit from input lane impacts several bits in output accumulator. All operations are performed modulo 2^64. Input is consumed one full stripe at a time. Step 2 is looped as many times as necessary to consume the whole input, except for the last remaining bytes which cannot form a stripe (< 32 bytes). When that happens, move to step 3.

Step 3. Accumulator convergence All 4 lane accumulators from previous steps are merged to produce a single remaining accumulator of same width (64-bit). The associated formula is as follows. Note that accumulator convergence is more complex than 32-bit variant, and requires to define another function called mergeAccumulator():

which is then used in the convergence formula:

Step 4. Add input length The input total length is presumed known at this stage. This step is just about adding the length to accumulator, so that it participates to final mixing.

Step 5. Consume remaining input There may be up to 31 bytes remaining to consume from the input. The final stage will digest them according to following pseudo-code:

= 8) { lane = read_64bit_little_endian(input_ptr); acc = acc xor round(0, lane); acc = (acc <<< 27) * PRIME64_1; acc = acc + PRIME64_4; input_ptr += 8; remainingLength -= 8; } if (remainingLength >= 4) { lane = read_32bit_little_endian(input_ptr); acc = acc xor (lane * PRIME64_1); acc = (acc <<< 23) * PRIME64_2; acc = acc + PRIME64_3; input_ptr += 4; remainingLength -= 4; } while (remainingLength >= 1) { lane = read_byte(input_ptr); acc = acc xor (lane * PRIME64_5); acc = (acc <<< 11) * PRIME64_1; input_ptr += 1; remainingLength -= 1; } ]]> This process ensures that all input bytes are present in the final mix.

> 33); acc = acc * PRIME64_2; acc = acc xor (acc >> 29); acc = acc * PRIME64_3; acc = acc xor (acc >> 32); ]]>

Step 7. Output The XXH64() function produces an unsigned 64-bit value as output. For systems which require to store and/or display the result in binary or hexadecimal format, the canonical format is defined to reproduce the same value as the natural decimal format, hence follows big-endian convention (most significant byte first).

XXH3 Algorithm Overview XXH3 comes in two different versions: XXH3-64 and XXH3-128 (or XXH128), producing 64 and 128 bits of output, respectively. XXH3 uses different algorithms for small (0-16 bytes), medium (17-240 bytes), and large (241+ bytes) inputs. The algorithms for small and medium inputs are optimized for performance. The three algorithms are described in the following sections. Many operations require some 64-bit prime number constants, which are mostly the same constants used in XXH32 and XXH64, all defined below:

The XXH3_64bits() function produces an unsigned 64-bit value. The XXH3_128bits() function produces a XXH128_hash_t struct containing low64 and high64 - the lower and higher 64-bit half values of the result, respectively. For systems requiring storing and/or displaying the result in binary or hexadecimal format, the canonical format is defined to reproduce the same value as the natural decimal format, hence following big-endian convention (most significant byte first).

Seed and Secret XXH3 provides seeded hashing by introducing two configurable constants used in the hashing process: the seed and the secret. The seed is an unsigned 64-bit value, and the secret is an array of bytes that is at least 136 bytes in size. The default seed is 0, and the default secret is the following 192-byte value:

The seed and the secret can be optionally specified using the *_withSecret and *_withSeed versions of the hash function. The seed and the secret cannot be specified simultaneously (*_withSecretAndSeed is actually *_withSeed for short and medium inputs <= 240 bytes, and *_withSecret for large inputs). When one is specified, the other one uses the default value. There is one exception though: when input is large (> 240 bytes) and a seed is given, a secret is derived from the seed value and the default secret using the following procedure:

The derivation treats the secrets as 24 64-bit values. In XXH3 algorithms, the secret is always read similarly by treating a contiguous segment of the array as one or more 32-bit or 64-bit values. The secret values are always read using little-endian convention.

Final Mixing Step (avalanche) To make sure that all input bits have a chance to impact any bit in the output digest (avalanche effect), the final step of the XXH3 algorithm is usually one of the two fixed operations that mix the bits in a 64-bit value. These operations are denoted avalanche() and avalanche_XXH64() in the following XXH3 description.

> 37); x = x * PRIME_MX1; x = x xor (x >> 32); return x; avalanche_XXH64(u64 x): x = x xor (x >> 33); x = x * PRIME64_2; x = x xor (x >> 29); x = x * PRIME64_3; x = x xor (x >> 32); return x; ]]>

XXH3 Algorithm Description (for small inputs) The algorithm for small inputs (0-16 bytes of input) is further divided into 4 cases: empty, 1-3 bytes, 4-8 bytes, and 9-16 bytes of input. The algorithm uses byte-swap operations. The byte-swap operation reverses the byte order in a 32-bit or 64-bit value. It is denoted bswap32 and bswap64 for its 32-bit and 64-bit versions, respectively.

Empty input The hash of empty input is calculated from the seed and a segment of the secret:

1-3 bytes of input The algorithm starts from a single 32-bit value combining the input bytes and its length:

>1] << 24); // LSB 8 16 24 MSB // | last byte | length | first byte | middle-or-last byte | ]]> Then the final output is calculated from the value and the first 8 bytes (XXH3-64) or 16 bytes (XXH3-128) of the secret to produce the final result. The secret here is read as 32-bit values instead of the usual 64-bit values.

Note that the XXH3-64 result is the lower half of XXH3-128 result.

4-8 bytes of input The algorithm starts from reading the first and last 4 bytes of the input as little-endian 32-bit values, and a modified seed:

Again, these values are combined with a segment of the secret to produce the final value.

> 35) + inputLength); value = value * PRIME_MX2; value = value xor (value >> 28); return value; XXH3_128_4to8(): u64 secretWords[2] = secret[16:32]; u64 combined = (u64)inputFirst | ((u64)inputLast << 32); u64 value = ((secretWords[0] xor secretWords[1]) + modifiedSeed) xor combined; u128 mulResult = (u128)value * (u128)(PRIME64_1 + (inputLength << 2)); u64 high = higherHalf(mulResult); // mulResult >> 64 u64 low = lowerHalf(mulResult); // mulResult & 0xFFFFFFFFFFFFFFFF high = high + (low << 1); low = low xor (high >> 3); low = low xor (low >> 35); low = low * PRIME_MX2; low = low xor (low >> 28); high = avalanche(high); return {low, high}; ]]>

9-16 bytes of input The algorithm starts from reading the first and last 8 bytes of the input as little-endian 64-bit values:

Once again, these values are combined with a segment of the secret to produce the final value.

128 multiplication ({low,high} = (u128){low,high} * PRIME64_2) u128 mulResult2 = (u128)low * (u128)PRIME64_2; low = lowerHalf(mulResult2); high = higherHalf(mulResult2) + high * PRIME64_2; return {avalanche(low), // lower half avalanche(high)}; // higher half ]]>

XXH3 Algorithm Description (for medium inputs) This algorithm is used for medium inputs (17-240 bytes of input). Its internal hash state is stored inside 1 (XXH3-64) or 2 (XXH3-128) "accumulators", each storing an unsigned 64-bit value.

Step 1. Initialize internal accumulators The accumulator(s) are initialized based on the input length.

Step 2. Process the input This step is further divided into two cases: one for 17-128 bytes of input, and one for 129-240 bytes of input.

Mixing operation This step uses a mixing operation that mixes a 16-byte segment of data, a 16-byte segment of secret and the seed into a 64-bit value as a building block. This operation treat the segment of data and secret as little-endian 64-bit values.

The mixing operation is always invoked in groups of two in XXH3-128, where two 16-byte segments of data are mixed with a 32-byte segment of secret, and the accumulators are updated accordingly.

The input is split into several 16-byte chunks and mixed, and the result is added to the accumulator(s).

17-128 bytes of input The input is read as N 16-byte chunks starting from the beginning and N chunks starting from the end, where N is the smallest number that these 2N chunks cover the whole input. These chunks are paired up and mixed, and the results are accumulated to the accumulator(s).

> 5) + 1; for (i = numRounds - 1; i >= 0; i--) { size offsetStart = i*16; size offsetEnd = inputLength - i*16 - 16; acc += mixStep(input[offsetStart:offsetStart+16], i*32, seed); acc += mixStep(input[offsetEnd:offsetEnd+16], i*32+16, seed); } processInput_XXH3_128_17to128(): u64 numRounds = ((inputLength - 1) >> 5) + 1; for (i = numRounds - 1; i >= 0; i--) { size offsetStart = i*16; size offsetEnd = inputLength - i*16 - 16; mixTwoChunks(input[offsetStart:offsetStart+16], input[offsetEnd:offsetEnd+16], i*32, seed); } ]]>

129-240 bytes of input The input is split into 16-byte (XXH3-64) or 32-byte (XXH3-128) chunks. The first 128 bytes are first mixed chunk by chunk, followed by an intermediate avalanche operation. Then the remaining full chunks are processed, and finally the last 16/32 bytes are treated as a chunk to process.

> 4; for (i = 0; i < 8; i++) { acc += mixStep(input[i*16:i*16+16], i*16, seed); } acc = avalanche(acc); for (i = 8; i < numChunks; i++) { acc += mixStep(input[i*16:i*16+16], (i-8)*16 + 3, seed); } acc += mixStep(input[inputLength-16:inputLength], 119, seed); processInput_XXH3_128_129to240(): u64 numChunks = inputLength >> 5; for (i = 0; i < 4; i++) { mixTwoChunks(input[i*32:i*32+16], input[i*32+16:i*32+32], i*32, seed); } acc[0] = avalanche(acc[0]); acc[1] = avalanche(acc[1]); for (i = 4; i < numChunks; i++) { mixTwoChunks(input[i*32:i*32+16], input[i*32+16:i*32+32], (i-4)*32 + 3, seed); } // note that the half-chunk order and the seed is different here mixTwoChunks(input[inputLength-16:inputLength], input[inputLength-32:inputLength-16], 103, (u64)0 - seed); ]]>

Step 3. Finalization The final result is extracted from the accumulator(s).

XXH3 Algorithm Description (for large inputs) This algorithm is used for inputs larger than 240 bytes. The internal hash state is stored inside 8 "accumulators", each one storing an unsigned 64-bit value.

Step 1. Initialize internal accumulators The accumulators are initialized to fixed constants:

Step 2. Process blocks The input is consumed and processed one full block at a time. The size of the block depends on the length of the secret. Specifically, a block consists of several 64-byte stripes. The number of stripes per block is floor((secretLength-64)/8) . For the default 192-byte secret, there are 16 stripes in a block, and thus the block size is 1024 bytes.

The process of processing a full block is called a round. It consists of the following two sub-steps:

Step 2-1. Process stripes in the block A stripe is evenly divided into 8 lanes, of 8 bytes each. In an accumulation step, one stripe and a 64-byte contiguous segment of the secret are used to update the accumulators. Each lane reads its associated 64-bit value using little-endian convention. The accumulation step applies the following procedure:

> 32) } ]]> The accumulation step is repeated for all stripes in a block, using different segments of the secret, starting from the first 64 bytes for the first stripe, and offset by 8 bytes for each following round:

Step 2-2. Scramble accumulators After the accumulation steps are finished for all stripes in the block, the accumulators are scrambled using the last 64 bytes of the secret.

> 47); acc[i] = acc[i] xor secretWords[i]; acc[i] = acc[i] * PRIME32_1; } ]]> A round is thus a round_accumulate followed by a round_scramble:

Step 2 is looped to consume the input until there are less than or equal to blockSize bytes of input left. Note that we leave the last block to the next step even if it is a full block.

Step 3. Process the last block and the last 64 bytes Accumulation steps are run for the stripes in the last block, except for the last stripe (whether it is full or not). After that, run a final accumulation step by treating the last 64 bytes as a stripe. Note that the last 64 bytes might overlap with the second-to-last block.

Step 4. Finalization In the finalization step, a merging procedure is used to extract a single 64-bit value from the accumulators, using an initial seed value and a 64-byte segment of the secret.

> 64) } return avalanche(result); ]]> XXH3-128 runs the merging procedure twice for the two halves of the result, using different secret segments and different initial values derived from the total input length. The XXH3-64 result is just the lower half of the XXH3-128 result.

Performance considerations The xxHash algorithms are simple and compact to implement. They provide a system independent "fingerprint" or digest of a message of arbitrary length. The algorithm allows input to be streamed and processed in multiple steps. In such case, an internal buffer is needed to ensure data is presented to the algorithm in full stripes. On 64-bit systems, the 64-bit variant XXH64 is generally faster to compute, so it is a recommended variant, even when only 32-bit are needed. On 32-bit systems though, positions are reversed: XXH64 performance is reduced, due to its usage of 64-bit arithmetic. XXH32 becomes a faster variant. Finally, when vector operations are possible, XXH3 is likely the faster variant.

Reference Implementation A reference library written in C is available at https://www.xxhash.com . The web page also links to multiple other implementations written in many different languages. It links to the github project page where an issue board can be used for further public discussions on the topic.

IANA Considerations None

Security Considerations xxHash is not a cryptographic hash function, and has not been designed for use in any cryptographic setting. Implementations has to follow best practices to avoid security concerns, and users needs to continously re-evaulate implementations for security vulnerabilities.

Notices This document is derived from https://github.com/Cyan4973/xxHash/blob/dev/doc/xxhash_spec.md commit d66a9cb6f223b1f15cceebee39bc07ed0bd4ad3d with notice below, with all changes are tracked at https://gitlab.com/jas/ietf-xxhash using Git.