single mention · not yet filed
Document-level stream processing in the training data loader
Document-level stream processing splits the training document stream into disjoint per-rank and per-worker slices, each running its own pipeline.1 The pipeline unrolls raw text into individual documents, tokenises each online, splits long ones into chunks, and bin-packs the chunks into full-length sequences.2
Trade-off Unlike concat-and-chunk packing, it avoids cutting documents that would fit into one sequence.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
each rank sees a disjoint slice of the index and each worker of a rank a disjoint slice of that, and every worker then runs the same per-worker pipeline
— tech reportunrolls them into individual documents. 3. tokenises each document online
— tech reportThis concat- and-chunk packing has two problems. First, it cuts documents that would fit into one sequence
— tech report
Also called unrolls them into individual documents.