Open Models Almanac
Methodsdata curationsequence packing

single source · data curation

Random Sequential Document Buffer (RSDB)

A Random Sequential Document Buffer breaks sequence-packing locality during data loading by shuffling document chunks.1 Each worker keeps a buffer of documents with read offsets and emits, at each step, a random document's next contiguous chunk, refilling slots as documents finish.2

Trade-off It keeps per-step batches closer to the dataset distribution while preserving within-document order, and is reported to smooth the training loss.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. We break this locality during data loading with anRSDB — tech report
  2. Each dataloader worker keeps a buffer of 4096 tokenised documents together with a read offset per document. T o produce the next piece, the worker draws a buffered document uniformly at random, emits the next contiguous chunk of at most one sequence length starting at that document’s offset — tech report
  3. Within a document the chunks keep their original order; only their spacing in the training stream changes. V. Singh et al. (2026) report a smoother training loss with the RSDB than with sequential chunking — tech report

Also called Random Sequential Document Buffer.

Filed under data curation › sequence packing