LFM3-1.7B
Liquid AIWe perform exact deduplication through direct matching
specific method · filed under data curation
Exact text deduplication removes every document whose raw UTF-8 text hashes identically to another document's.1 It hashes each document, compares hashes within and across snapshots in chronological order, and keeps the earliest copy.2
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
Exact deduplication removes every document whose UTF-8 text, without normalisation, has the same 128- bit xxHash9 as another document— tech report
first within each snapshot and then across all snapshots in chronological order, so the earliest copy remains— tech report
Also called Exact document deduplication, Exact deduplication, Exact-match deduplication.
2 quotes from 2 sources, one card per model, each opening with its strongest.
We perform exact deduplication through direct matching
Exact deduplication removes byte-identical documents across all snapshots