lukechampine.com/lthash

lthash

go get lukechampine.com/lthash

This repo contains an implementation of LtHash, as defined by Bellare and Micciancio and later specified more concretely by researchers at Facebook.

LtHash is a homomorphic hash function based on BLAKE2x. A homomorphic hash function provides a solution to the following problem: Given the hash of an input, along with a small update to the input, how can we compute the hash of the new input with its update applied, without having to recompute the entire hash from scratch?

For example, say you have a database that contains three items: "Apple", "Banana", and "Orange". We compute the LtHash of this set by summing the hashes of the individual elements. Later, we replace "Banana" with "Grape". To compute the new hash, we take our original hash, subtract the hash of "Banana", and add the hash of "Grape". The resulting hash is the same as if the database originally contained "Grape" instead of "Banana".

This repo currently contains an implementation of lthash16. Facebook's paper further specifies lthash20 and lthash32; these may be added in the future.

Be aware that LtHash is vulnerable to multiset input collisions. A multiset is a set containing more than one instance of a particular element. In particular, it is trivial to produce a collision in lthash16 by adding the same input to the hash 2^16 times. One way to prevent this is to concatenate each input with a unique piece of metadata, such as an index.

Usage

h := lthash.New16()

// compute the combined hash of "Apple", "Banana", and "Orange"
h.Add([]byte("Apple"))
h.Add([]byte("Banana"))
h.Add([]byte("Orange"))
oldSum := h.Sum(nil)

// replace "Banana" with "Grape"; the resulting hash is the same
// as the combined hash of "Apple", "Grape", and "Orange".
h.Remove([]byte("Banana"))
h.Add([]byte("Grape"))
newSum := h.Sum(nil)

Benchmarks

Tested on an i5-7600K @ 3.80GHz. Results will likely be slower on non-amd64 architectures.

BenchmarkLtHash/16-4    200000    9651 ns/op    424.38 MB/s    0 allocs/op