single mention · data curation
Exact text deduplication by xxHash
Exact deduplication removes documents whose unnormalised UTF-8 text hashes to the same 128-bit xxHash as another document.1 It first deduplicates within each snapshot, then across snapshots in chronological order so the earliest copy is kept.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
Exact deduplication removes every document whose UTF-8 text, without normalisation, has the same 128- bit xxHash9 as another document
— tech reportfirst within each snapshot and then across all snapshots in chronological order, so the earliest copy remains
— tech report
Also called Exact deduplication.
Filed under data curation › deduplication