Open Models Almanac
Methodsdata curationdeduplication

single source · data curation

Substring deduplication

Substring deduplication is a data-curation step that removes repeated byte sequences within documents, keeping one occurrence of each repeated block.1 It uses a suffix-array method to find the repeats, targeting boilerplate such as navigation menus and page templates.2,3

Trade-off It removes a large share of text bytes while affecting document counts only slightly, since it cuts text within documents rather than dropping whole documents.4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. keeps one occurrence of every repeated byte sequence of at least 500 bytes and cuts the others out of their documents — tech report
  2. Substring deduplication follows the suffix-array method of Lee et al. (2022 — tech report
  3. such as navigation menus and page templates — tech report
  4. Substring deduplication removes text within documents and drops a document only when nothing of it remains, so its effect on bytes is larger than on documents — tech report

Also called Suffix-array substring deduplication.

Filed under data curation › deduplication