Open Models Almanac
Methodsdata curationdeduplication

methods · data curation

deduplication

15 methods filed at this node or below it, from the sources of 6 models.

In this branch 15

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 15

Corpus deduplication used · 5 sources · 5 quotes
Exact text deduplication used · 2 sources · 2 quotes
13 single mentions
CMinHash locality-sensitive hashing used · 1 source · 1 quote
Exact question–answer deduplication used · 1 source · 1 quote
Exact text deduplication by xxHash used · 1 source · 1 quote
In-pack deduplication used · 1 source · 1 quote
Retain-one substring removal used · 1 source · 1 quote
Semantic deduplication used · 1 source · 1 quote
Snapshot-level fuzzy deduplication used · 1 source · 1 quote
Substring deduplication used · 1 source · 2 quotes
Token 3-gram Jaccard verification used · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

↑