Open Models Almanac
Methodsdata curationdeduplication

single mention · data curation

Retain-one substring removal

Keeps one occurrence of each repeated byte sequence of at least 500 bytes and cuts other occurrences out of their documents.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See deduplication for the methods filed beside it.

Filed under data curation › deduplication