single source · data curation
Random Sequential Document Buffer (RSDB)
A Random Sequential Document Buffer breaks sequence-packing locality during data loading by shuffling document chunks.1 Each worker keeps a buffer of documents with read offsets and emits, at each step, a random document's next contiguous chunk, refilling slots as documents finish.2
Trade-off It keeps per-step batches closer to the dataset distribution while preserving within-document order, and is reported to smooth the training loss.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
We break this locality during data loading with anRSDB
— tech reportEach dataloader worker keeps a buffer of 4096 tokenised documents together with a read offset per document. T o produce the next piece, the worker draws a buffered document uniformly at random, emits the next contiguous chunk of at most one sequence length starting at that document’s offset
— tech reportWithin a document the chunks keep their original order; only their spacing in the training stream changes. V. Singh et al. (2026) report a smoother training loss with the RSDB than with sequential chunking
— tech report
Also called Random Sequential Document Buffer.
Filed under data curation › sequence packing