Open Models Almanac
Methodsdata curationdata sourcing

single mention · data curation

Long-context training-data construction

Long-context training-data construction involves curating datasets with long-context documents to enable reasoning over large documents.1 This is achieved by length-bucketing the training pool and blending it with OCR'd PDFs, which contain documents up to 256k tokens.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we curated datasets that contained long-context documents based on publicly available data (e.g., from German legislative documents) with corresponding question/answering prompts — Aleph-Alpha/Kolibri-1 · Hugging Face
  2. Long-context candidates were built by length-bucketing the mid-training pool and blending it with OCR'd PDFs — Aleph-Alpha/Kolibri-1 · Hugging Face

Also called Long-context.

Filed under data curation › data sourcing