Open Models Almanac
Methods

single mention · not yet filed

Document-level stream processing in the training data loader

Document-level stream processing splits the training document stream into disjoint per-rank and per-worker slices, each running its own pipeline.1 The pipeline unrolls raw text into individual documents, tokenises each online, splits long ones into chunks, and bin-packs the chunks into full-length sequences.2

Trade-off Unlike concat-and-chunk packing, it avoids cutting documents that would fit into one sequence.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. each rank sees a disjoint slice of the index and each worker of a rank a disjoint slice of that, and every worker then runs the same per-worker pipeline — tech report
  2. unrolls them into individual documents. 3. tokenises each document online — tech report
  3. This concat- and-chunk packing has two problems. First, it cuts documents that would fit into one sequence — tech report

Also called unrolls them into individual documents.