single mention · data curation
Retain-one substring removal
Keeps one occurrence of each repeated byte sequence of at least 500 bytes and cuts other occurrences out of their documents.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See deduplication for the methods filed beside it.
Filed under data curation › deduplication