Open Models Almanac
Methodsdata curationdeduplication

single source · data curation

MinHash locality-sensitive hashing fuzzy deduplication

Fuzzy deduplication uses MinHash (Broder1997) with locality-sensitive hashing.1 Documents are normalized and shingled into token 5-grams to create signatures that pair documents based on Jaccard similarity.2,3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. Fuzzy deduplication uses MinHash (Broder1997) with locality-sensitive hashing — tech report
  2. Documents are normalised as in RefinedWeb and CCNet (Wenzek et al.2020), with every digit replaced by a placeholder, and shingled into token 5-grams — tech report
  3. The signatures have 26 bands of 11 rows, which pair two documents with probability one half at a Jaccard similarity of 0.72 and 99 % at 0.85 — tech report

Also called Fuzzy deduplication.

Filed under data curation › deduplication