single source · data curation
Substring deduplication
Substring deduplication is a data-curation step that removes repeated byte sequences within documents, keeping one occurrence of each repeated block.1 It uses a suffix-array method to find the repeats, targeting boilerplate such as navigation menus and page templates.2,3
Trade-off It removes a large share of text bytes while affecting document counts only slightly, since it cuts text within documents rather than dropping whole documents.4
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
keeps one occurrence of every repeated byte sequence of at least 500 bytes and cuts the others out of their documents
— tech reportSubstring deduplication follows the suffix-array method of Lee et al. (2022
— tech reportsuch as navigation menus and page templates
— tech reportSubstring deduplication removes text within documents and drops a document only when nothing of it remains, so its effect on bytes is larger than on documents
— tech report
Also called Suffix-array substring deduplication.
Filed under data curation › deduplication