Open Models Almanac
Methodsdata curationdeduplication

specific method · filed under data curation

Exact text deduplication

Exact text deduplication removes every document whose raw UTF-8 text hashes identically to another document's.1 It hashes each document, compares hashes within and across snapshots in chronological order, and keeps the earliest copy.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. Exact deduplication removes every document whose UTF-8 text, without normalisation, has the same 128- bit xxHash9 as another document — tech report
  2. first within each snapshot and then across all snapshots in chronological order, so the earliest copy remains — tech report

Also called Exact document deduplication, Exact deduplication, Exact-match deduplication.

sources
2
models
2
labs adopt it
2
strongest
used

Documented in

Aleph Alpha
Liquid AI

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 2

Kolibri-1

Aleph Alphaadopted1 source

Exact deduplication removes byte-identical documents across all snapshots

usedunclearin KolibriAleph Alpha
↑