Open Models Almanac
ModelsKolibri-1

Aleph Alpha

Kolibri-1

What Kolibri-1’s own documents say it is built from — every method with the sentence that describes it.

Parameters
78B / 3.46B active
Context
256K
Attention
sliding/global hybrid4:1 sliding-window (513) : global GQA, positional encoding only in SWA layers (NoPE global)
Experts
384 experts: 1 shared + 6 routed active
Input
text
Licence
Apache 2.0
Released
2026-10-03

Against the consensus recipe

Its documents state 9 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 9 of the 19 architecture features checked — 3 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_kolibri1.py, configuration_kolibri1.py) and config.json at revision abfe85f673, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention code only in its codenot stated

48 query heads share 4 key/value heads via repeat_kv in every attention layer.

show evidencenum_attention_heads = 48num_key_value_heads = 4
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_kolibri1.py · L145
    key_states = repeat_kv(key, module.num_key_value_groups)
    value_states = repeat_kv(value, module.num_key_value_groups)
modeling_kolibri1.py · L121–122
Multi-head latent attention not in its codenot stated

Attention uses standard separate q/k/v projections with no low-rank latent cache or kv_lora_rank, so MLA is not implemented.

Sliding-window attention in its codeadopted

use_sliding_window is true and sliding_window=513, and layers tagged sliding_attention apply this window via…

show evidence

use_sliding_window is true and sliding_window=513, and layers tagged sliding_attention apply this window via create_sliding_window_causal_mask.

use_sliding_window = truesliding_window = 513
        self.sliding_window = config.sliding_window
        layer_type = config.layer_types[layer_idx] if hasattr(config, "layer_types") else None
        self.is_sliding = layer_type == "sliding_attention"
modeling_kolibri1.py · L149–151
Interleaved sliding-window and global attention in its codeadopted

layer_types interleaves four sliding_attention layers followed by one full_attention layer, repeating across all 50 layers.

show evidencelayer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "sliding_atte…
            causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "sliding_attention": create_sliding_window_causal_mask(**mask_kwargs),
            }
modeling_kolibri1.py · L498–501
Indexer-selected sparse attention (DeepSeek Sparse Attention) not in its codenot stated

No indexer scoring module or top-k token selection mechanism exists in the attention code; config has no index_topk/index_n_heads keys.

Linear-attention or state-space layers alongside full attention not in its codenot stated

All decoder layers use Kolibri1Attention (softmax attention with sliding or full mask); no recurrent/linear/state-space mixer layer type…

show evidence

All decoder layers use Kolibri1Attention (softmax attention with sliding or full mask); no recurrent/linear/state-space mixer layer type is present.

        self.self_attn = Kolibri1Attention(config=config, layer_idx=layer_idx)
        self.mlp = Kolibri1SparseMoeBlock(config)
modeling_kolibri1.py · L321–322
Gated DeltaNet layers not in its codenot stated

No delta-rule recurrent update or decay gate is implemented anywhere in the modeling file.

Mamba-2 layers not in its codenot stated

No SSM/Mamba-2 state-space block or related config keys (mamba_num_heads, ssm_state_size) appear in this model.

Learnable attention sink not in its codenot stated

The attention softmax has no extra learned sink logit column; eager_attention_forward only computes query-key-value softmax.

show evidence
    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
modeling_kolibri1.py · L128
Gated attention output not in its codenot stated

Attention output goes directly from attn_output reshape to o_proj with no sigmoid gating multiplication applied.

show evidence
        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights
modeling_kolibri1.py · L204–206
QK normalization in its codeadopted

Queries and keys are each passed through a per-head RMSNorm (q_norm/k_norm) before rotary embedding and attention.

show evidence
        self.q_norm = Kolibri1RMSNorm(self.head_dim, eps=config.rms_norm_eps)
        self.k_norm = Kolibri1RMSNorm(self.head_dim, eps=config.rms_norm_eps)
modeling_kolibri1.py · L158–159
        query_states = self.q_norm(query_states)
        key_states = self.k_norm(key_states)
modeling_kolibri1.py · L177–178
Partial RoPE not in its codenot stated

RoPE is computed over the full head_dim (rotate_half splits the whole dimension in half for rotation, not a partial subset), with no…

show evidence

RoPE is computed over the full head_dim (rotate_half splits the whole dimension in half for rotation, not a partial subset), with no partial_rotary_factor or rope_dim config present.

        base = config.rope_parameters["rope_theta"]
        dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads
modeling_kolibri1.py · L423–424
YaRN RoPE scaling code unclearnot stated

rope_parameters is read from config but its contents (rope_type, possible yarn scaling) are not shown in the given config.json, which only…

show evidence

rope_parameters is read from config but its contents (rope_type, possible yarn scaling) are not shown in the given config.json, which only lists rope_theta at top level.

rope_theta = 10000.0
        self.rope_type = self.config.rope_parameters["rope_type"]
        rope_init_fn: Callable = self.compute_default_rope_parameters
        if self.rope_type != "default":
            rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
modeling_kolibri1.py · L403–406
Layers without positional encoding (NoPE) in its codeadopted

Full-attention layers skip rotary position embedding entirely (NoPE) while sliding-window layers apply RoPE, per the is_sliding check.

show evidencelayer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "sliding_atte…
        # We use global NoPE for hybrid attention model
        if self.sliding_window is None or self.is_sliding:
            query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
modeling_kolibri1.py · L181–183
Mixture of experts in its codeadopted

Each token is routed via Kolibri1TopKRouter to 6 of 384 expert MLPs in Kolibri1SparseMoeBlock.

show evidencenum_experts = 384num_experts_per_tok = 6
        router_indices = torch.topk(router_logits + self.e_score_correction_bias, self.top_k, dim=-1)[1]
        router_scores = router_logits.gather(1, router_indices).sigmoid()
modeling_kolibri1.py · L288–289
Shared expert in its codeadopted

A shared_experts MLP with nonzero intermediate size (512) is added to every token's output alongside routed experts.

show evidenceshared_expert_intermediate_size = 512
        self.shared_experts = Kolibri1MLP(config, intermediate_size=config.shared_expert_intermediate_size)
modeling_kolibri1.py · L305
        hidden_states = hidden_states + self.shared_experts(residuals)
modeling_kolibri1.py · L313
Auxiliary-loss-free load balancing (selection bias) code only in its codenot stated

The router adds a learned e_score_correction_bias to logits only for top-k expert selection, not for the output weighting (which uses raw…

show evidence

The router adds a learned e_score_correction_bias to logits only for top-k expert selection, not for the output weighting (which uses raw sigmoid logits).

        self.e_score_correction_bias = nn.Buffer(torch.zeros(self.num_experts))
modeling_kolibri1.py · L273
        router_indices = torch.topk(router_logits + self.e_score_correction_bias, self.top_k, dim=-1)[1]
        router_scores = router_logits.gather(1, router_indices).sigmoid()
modeling_kolibri1.py · L288–289
Multi-token prediction layers not in its codenot stated

No multi-token-prediction modules or layer count config (num_nextn_predict_layers, mtp_num_layers) appear in the config or code.

SwiGLU feed-forward code only in its codenot stated

Both the shared-expert MLP and the routed experts compute silu(gate(x)) * up(x) as hidden_act is silu.

show evidencehidden_act = "silu"
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
modeling_kolibri1.py · L221
            gate, up = nn.functional.linear(current_state, self.gate_up_proj[expert_idx]).chunk(2, dim=-1)
            current_hidden_states = self.act_fn(gate) * up
modeling_kolibri1.py · L256–257

token mixer 3

softmax attention 2

global attention 1

Full attention without positional encoding core

“the other 10 use full attention without positional encoding”

sliding window attention 1

Sliding Window Attention core

“restrict attention to a fixed-size window of preceding tokens, reducing both memory usage and computational cost”

hybrid layer stacking 1

Hybrid Sliding-Window and Global Attention core

“Kolibri uses full attention in every fifth block (10 of 50) and SWA in the other 40.”

channel mixer 9

mixture of experts 9

Mixture of experts core

“Kolibri is the second model in the Kolibri family, an English–German MoE”

expert routing 1

Top-k expert routing core

“each MoE layer routes a token to 6 of its 384 routed experts”

expert load balancing 5

Exact Quantile Balancing core

“Expert loads are balanced with Exact Quantile Balancing (EQB) and Load-Error Injection (LEI)”

Load-error gradient injection core

“Expert loads are balanced with Exact Quantile Balancing (EQB) and Load-Error Injection (LEI)”

Mixture-of-experts load balancing used

“To balance the workload across all expert parallel ranks, we rely on our MoE load balancing methodology as described in Section 2.1.5.”

Recentered global-quantile target update used

“The expert-balancing bias was updated separately using a recentered global-quantile target update.”

Two-pass radix selection used

“EQB (Neitemeier et al. 2026) computes this global quantile exactly for bfloat16 adjusted logits using two-pass radix selection”

shared experts 1

Shared expert core

“in addition to one shared expert.”

fine-grained experts 1

Many small experts used

“We used 384 smaller experts rather than fewer wide ones, as they performed better in our tests.”

positional encoding 2

No Position Encoding core

“the full-attention blocks use No Positional Encoding (NoPE)”

Static YaRN used

“Qwen3.5 with static YaRN at factor 4 at every length”

normalization & residual 1

QK Normalization core

“Queries and keys are normalised per head with RMS normalisation (Henry et al. 2020; B. Zhang and Sennrich 2019) before RoPE”

context capacity 1

Context-length extrapolation without positional rescaling used

“evaluation beyond the training length needs no positional rescaling”

7 more mentioned or evaluated
Expert granularity evaluated

fine-grained experts “We compare three expert configurations of about 80B total parameters”

QK-Clip evaluated

normalization & residual “An adapted QK-Clip (Kimi Team 2025) for GQA with threshold 100 in place of QK normalisation also shows gradient-norm spikes.”

Coarse-grained MoE configuration evaluated

fine-grained experts “with width 2048 and 6 of 256 experts of hidden dimension 1024”

Fine-grained MoE configuration evaluated

fine-grained experts “with width 2048 and 12 of 512 experts of the same hidden dimension”

Random routed-expert selection evaluated

expert routing “Random routed experts replaces each token’s selected experts with a uniform random draw and tests whether the routing decision matters”

Residual-stream width selection evaluated

“We compare residual widths of 640, 1024 and 1600 at a fixed depth of 50 layers.”

Routed-expert silencing test evaluated

mixture of experts “silenced routed experts removes the routed output, keeping the shared expert, and tests whether the routed experts contribute at all.”

Not used 2

Leading dense feed-forward layers before MoE layers not used

dense feed-forward network “Kolibri Origin uses dense FFNs in its first two blocks”

MoE layer densification by expert concatenation not used

mixture of experts “We tried fixing this issue during SFT, where load balancing is disabled, through densifying layers 0 and 1 by concatenating the shared…”

language modelling objective 1

Causal next-token prediction core

“pretrained the model with a causal next-token-prediction objective on a large and diverse document corpus”

auxiliary loss 3

Log-probability-ranked sentence masking core

“We approximate both masks by first ranking the sentences by how much masking each one individually changes the log-probability of a…”

Per-step policy-dependent recomputation of context masks core

“During training, we recompute the masks from the policy at every step”

Squared log-policy-ratio regularizer core

“We add a squared log-ratio term to softly penalise policy drift”

filed at the root 3

Model souping used

“each parameter of a soup is the mean of its values in the two models.”

Factorized hyperparameter transfer used

“Instead, we factorise the transfer across model size and token horizon”

Maximal Update Parametrization used

“For the width transfer we use Maximal Update Parametrisation (µP)”

optimizer 8

Muon used

“We optimise attention and expert weights with Muon”

AdamW used

“AdamW was used for the embedding matrix, one-dimensional backbone parameters, router weights, and language-model head.”

Category-specific optimizer assignment used

“it uses the optimiser groups of pre-training (Table 6), i.e., Muon (Jordan et al. 2024) for the attention and expert weights and Adam…”

Adam used

“embedding weights and norm scales with Adam”

Decoupled weight decay used

“We use decoupled weight decay for all optimiser groups except the embedding and norm scales.”

Learning-rate ratio tuning used

“we optimise the ratios between the learning rates of the different embedding, norm and head parameter groups as a factor of the Muon…”

Nesterov Muon used

“All other two-dimensional backbone parameters were optimized with Nesterov Muon using a learning rate of 0.001.”

Spectral scaling convention for Muon used

“we adopt the spectral convention ... which keeps the Muon learning rate independent of model width”

learning-rate schedule 2

Learning-rate tuning used

“we tune learning rates at the smallest ablation scale using η∈{η0/√2, η0, √2η0}”

Warmup-Stable-Merge used

“Kolibri is trained with a Warmup-Stable-Merge (WSM) recipe”

training precision 1

Mixed-precision distributed training with BF16 parameters and FP32 reductions used

“with bfloat16 parameters, float32 reductions, and a maximum gradient norm of 1.0.”

quantization-aware training 2

FP8 Quantization-Aware Training core

“The trainer therefore uses FP8 quantisation-aware training (Jacob et al. 2017), using the same quantisation as the inference engine.”

Quantization-Aware Training used

“additionally used quantization aware training to enable efficient low-precision inference with vLLM.”

training stability 3

Per-token logit-gradient norm bound core

“We therefore directly bound the norm by scaling our objective”

Global-norm gradient clipping used

“In addition to this per-token bound, we clip the global gradient norm.”

Per-token gradient-norm bounding used

“added a per-token gradient-norm bound (Section 3.2.1.2) after noticing that gradients for individual tokens reached into the thousands.”

training parallelism 4

Batch-size scaling with GPU count used

“As a remedy, we scale the batch size with the GPU count.”

Dedicated reduce-scatter process group used

“Dedicated NCCL process group for reduce-scatters, so they overlap with all-gathers instead of serialising behind them.”

Keep weights unsharded after backward across gradient-accumulation steps used

“Keep weights unsharded after the backward in-between the two gradient accumulation steps, removes the need to all-gather weights starting…”

Three-buffer reduce-scatter used

“Three reduce-scatter input buffers (about 800 MB each at this scale), to hide infrequent, non-deterministic communication latency that…”

training runtime 31

Version control as shared training-recipe consensus core

“Consensus comes from version control, where the main branch represents the team’s collective best understanding of how to train a model.”

Version-controlled model-development workflows core

“The Model Factory expresses these workflows as version-controlled code, which makes every experiment reproducible, composable, and…”

Version-controlled training pipeline and recipe core

“We implemented the training pipeline as code, so that the learnings of our team landed in one versioned training recipe instead of…”

Automated end-to-end training and evaluation on every code change default

“Every proposed code change triggers a small end-to-end model run; training, evaluation, to find out if something broke within minutes.”

Checkpointing every 250 training steps default

“We checkpoint every 250 training steps.”

Immutable version pinning for hermetic training runs default

“Every job pins versions of the code and of its artefacts (data, checkpoints, tokenisers), which we store immutably in a registry, so runs…”

Checkpoint merging used

“Checkpoint merging largely recovers this difference, with the four merged checkpoints indistinguishable within the standard error of the…”

Asynchronous checkpointing used

“The trainer asynchronously saves a checkpoint consisting of model weights, optimiser states and learning-rate schedules at regular…”

Automatic training recovery after failures used

“Our fault-tolerance setup combines frequent checkpointing with automatic recovery.”

Checkpointing complete data-loader state used

“its full state, from the position in the index down to the buffer contents of every worker, is checkpointed”

GPU-to-inference weight sync over GPUDirect RDMA used

“After every optimiser step, the trainer pushes updated weights to the inference replicas, as shown in Figure 37.”

Automatic faulty-node detection and replacement used

“Our cluster orchestration automatically detects faulty nodes and swaps them out of the main training run for automatic reboot and/or…”

Centralized block-runtime measurement collection used

“We explicitly gather the runtimes on GPU/process 0 to get more detailed statistics than just the average runtime.”

Checkpoint averaging used

“We then determine the base model weights as the average of the last 20 checkpoints of the long-context stage”

Cross-rank collective-kernel profile matching used

“we match the collective communication kernels between the profiles of all ranks compare their arrival time and observed throughput.”

CUDA-event timing of transformer blocks used

“time each transformer block during a forward pass with CUDA events”

Defer inter-replica all-reduce and FSDP2 finalization until the last microbatch used

“Microbatch-aware collectives that defer the inter-replica all-reduce and FSDP2’s end-of-backward finalisation to the last microbatch.”

Double-buffered CUDA-event evaluation used

“we use double-buffering. This means that each training step evaluates the CUDA events of its preceding step.”

Double-buffered RDMA batch transfer used

“It stages them in a double buffer, filling one half while the other is being sent.”

Frequent checkpointing for fault tolerance used

“Our fault-tolerance setup combines frequent checkpointing with automatic recovery.”

Full activation checkpointing used

“The trainer uses full activation checkpointing.”

Independent proxy-task scheduling to fill cluster gaps used

“each proxy training and evaluation task was queued independently such that it could fill the gaps between the larger jobs on the cluster.”

Node-local weight broadcast over NVLink used

“each receiving replica converts the weights in place into its kernel format and broadcasts them over NVLink to the other replicas of its…”

Periodic evaluation of merged recent checkpoints used

“Savanna registers each intermediate checkpoint as it appears, merges it with the preceding checkpoints of the configured window, and…”

Periodic profiling for straggler diagnosis used

“Our main training run writes a PyTorch profile once every 15 min”

PyTorch-profile-based straggler detection used

“For detecting complicated compute or communication straggler reasons, we use a profiling-based approach.”

Recompute expert dispatch during the backward pass used

“we re-use the same ElasticBuffer across all layers and recompute also the dispatch in the backward pass.”

Recompute MoE feed-forward activations instead of checkpointing them used

“To reduce device memory consumption, we avoid checkpointing any activation inside the MoE feed forward network.”

Reproducible training runs tied to version-control commits used

“Our runs are GitHub Actions workflows, and reproducing one means checking out a commit.”

Reuse one ElasticBuffer across MoE layers used

“To further reduce the memory consumption, we re-use the same ElasticBuffer across all layers”

Weight averaging of runs trained on different data mixes used

“combine resulting checkpoints by souping, which averages the weights of runs trained on different mixes into one model”

filed at the root 1

German web-data curation and synthetic data generation used

“we complement German datasets with more than 2T German tokens that we curate from the web with our own pipeline and generate synthetically”

data sourcing 4

Bilingual German-English data curation core

“Kolibri was trained on a filtered, bilingual (German/English) corpus combining curated web data, synthetic rephrasings and translations…”

Third-party data-source compliance screening core

“we assess Third-Party Datasets against the General Purpose AI Code of Practice commitments for direct crawling using all reasonably…”

Long-context training-data construction used

“Long-context candidates were built by length-bucketing the mid-training pool and blending it with OCR'd PDFs.”

Third-party data-source provenance screening used

“We assess the source of the data contained in the dataset on possible indications of copyright infringements”

data filtering 41

Benchmark decontamination core

“we remove from the training corpus every document that contains text from our evaluation suite.”

Shared canonical-format SFT data processing pipeline core

“A shared processing pipeline brings every sample into one canonical format, drops conversations that claim another model’s identity or…”

Personally identifiable information redaction used

“we redact PII from all training sources. Each category is detected by a regular expression, checked by a validator that rejects false…”

PII-redaction-threshold document filtering used

“A document whose redacted bytes exceed 15 % of its length is dropped.”

Averaging quality and grammar classifier scores used

“For German, we built upon our public work introduced in GermanWeb (T. F. Burns et al. 2026) by ensembling quality and grammar fastText…”

Benchmark reference-set coverage used

“The reference eval set contains all 226 benchmark tasks of our evaluation framework with all available splits (train, validation, and test)”

Bidirectional match-span expansion used

“On a hit, the inspection expands in both directions and keeps the evaluation instances that continue to match.”

Blind adjudication of disputed answer options used

“Kimi K2.6 rules on the disputed options without seeing the key.”

Candidate dataset audits used

“we check candidate datasets before they enter the pool. Sample audits of several candidate datasets found many completions of mixed quality”

Content-overlap filtering of candidate datasets used

“drop every candidate whose content largely overlaps another”

Dataset-pool cleanup, deduplication, and PII redaction used

“Every dataset of the pool passes the cleanup pass of Appendix D.1.1, exact deduplication across the whole pool (Appendix D.1.2), and PII…”

Difficulty-based filtering of long-context training questions used

“enters the training set only if GLM-5.2 solved it at least once and Kolibri at most six times”

Distilling LLM-judge annotation axes into binary classifiers used

“To score the rest of the crawl, we distil each scalar judgement axis into a binary task (top quartile versus bottom quartile, discarding…”

Equal-token-mass language-specific quality tiers used

“splitting each language into five equal-token-mass tiers.”

Exact-match requirement for short contamination spans used

“Short matches must be perfect, which keeps common phrases from being flagged.”

Excluding datasets with crawler opt-out violations used

“If we find indications that the entity crawling the data did not adhere to an appropriate opt-out, we exclude such datasets from our…”

Excluding datasets with source-level infringement indications used

“If we find indications for infringements on the data source level, we exclude the corresponding dataset from our model development.”

FastText classifier for SFT-style data quality used

“We combined the Luxical classifiers with a binary fastText classifier trained to distinguish SFT-style data vs. random CC samples”

Filtering synthetic generations for artifacts and errors used

“The outputs were post-processed and filtered to remove LLM-generation artefacts and erroneous generations.”

Five-word n-gram benchmark matching used

“We build on the decon engine of Olmo 3 (Allen Institute for AI 2025; Ettinger et al. 2025) and match word n-grams with n = 5.”

GermanWeb classifier scoring used

“the average of two GermanWeb classifiers for German”

Harmful and unsuitable training-data filtering used

“For English web data, keyword-based filtering additionally removed sexually explicit content.”

Heuristic filtering and language-based routing used

“Language-agnostic heuristic filters and language identification route each document to an English or a German branch, which apply separate…”

Heuristic language-consistency gate used

“Finally, a heuristic gate checks that both reasoning and answer are German, and an LLM language healer fixes the errors it finds in the…”

Heuristic web-document filtering and cleanup used

“Web documents passed a heuristic filter pipeline inspired by the approaches described in Dolma 3, DCLM and RefinedWeb”

Language-specific fastText routing and filtering used

“fastText language identification routing documents into an English and a German branch with language-specific thresholds”

Language-specific filter retuning used

“we had to retune filtering parameters for the German language.”

Language-specific web-data filtering used

“We therefore use FineWeb2’s German word-length settings, which allow a mean word length between 0 and 15 characters, instead of the 3 to…”

LLM-based language correction used

“Finally, a heuristic gate checks that both reasoning and answer are German, and an LLM language healer fixes the errors it finds in the…”

N-gram-based benchmark contamination detection used

“We detect contamination with the algorithm of the decon tool (Allen Institute for AI 2025).”

Pattern filtering of foreign model or lab identity claims used

“A pattern filter drops every conversation in which the assistant claims to be another model or to come from another lab, unless it also…”

Quality classification for quality-aware data mixing used

“Combined classifier scores were rescaled to [0, 1] in the annotation pipeline, and each language corpus was sorted into 20…”

Reference-answer filtering of generated solutions used

“We then reapply GLM-5.3 to solve the problem, and the answer is kept only if it matches the reference.”

Removing English reasoning from non-English SFT exchanges used

“For SFT, we therefore remove English reasoning from exchanges in other languages”

Removing synthetic long documents from the long-context mix used

“For the long-context mix, we removed synthetic long documents to avoid artificially inflating benchmarks such as RULER.”

Retaining answers while removing malformed or language-switching reasoning used

“A sample whose trace remains malformed or switches language keeps its answer without reasoning”

Simplex-weighted mixture of document annotation scores used

“We combine five English per-document scores with weights on the probability simplex.”

Test-based filtering of generated software-fix trajectories used

“Another GLM-5.3 instance then fixes the bug in the Terminus 2 harness of Terminal-Bench (Merrill et al. 2026), and we keep the…”

Training-data filtering and personal-data redaction used

“Our data pipeline filters against illegal, harmful, and pirated content and redacts personal data from data sources.”

URL-list filtering for unsafe or infringing sources used

“To prevent training on illegal, harmful, infringing or pirated content, we filtered our data based on a URL filter list.”

Within-benchmark inverse document frequency used

“We compute the IDF within each benchmark instead, so wording common to a benchmark counts for little.”

deduplication 10

Corpus deduplication used

“The Common Crawl corpus passed through exact/global (cross-dump) deduplication, MinHash fuzzy deduplication, and substring deduplication.”

Exact text deduplication used

“Exact deduplication removes byte-identical documents across all snapshots”

Chronological duplicate retention across snapshots used

“first within each snapshot and then across all snapshots in chronological order, so the earliest copy remains”

Digit-placeholder normalization and token 5-gram shingling used

“Documents are normalised as in RefinedWeb and CCNet (Wenzek et al.2020), with every digit replaced by a placeholder, and shingled into…”

Exact text deduplication by xxHash used

“Exact deduplication removes every document whose UTF-8 text, without normalisation, has the same 128-bit xxHash as another document”

Language-sharded chronological snapshot assignment used

“The stage runs per language in 30 English and 6 German shards, to which the snapshots are dealt round-robin in chronological order”

MinHash locality-sensitive hashing fuzzy deduplication used

“Fuzzy deduplication, which compares MinHash signatures of documents bucketed by locality-sensitive hashing, removes a further 25.9 % as…”

Retain-one substring removal used

“keeps one occurrence of every repeated byte sequence of at least 500 bytes and cuts the others out of their documents.”

Substring deduplication used

“Substring deduplication cuts repeated blocks of at least 500 bytes, such as navigation menus and page templates, out of the remaining…”

Token 3-gram Jaccard verification used

“every cluster of candidate pairs is verified on the Jaccard similarity of token 3-grams at a threshold of 0.80.”

synthetic data 30

Synthetic data generation used

“We produced synthetic data in English, using our deduplicated English CC data ... as seeds for rephrasing with Gemma-4-26B-A4B.”

German document rephrasing for data augmentation used

“We use Mistral-Nemo-Instruct-2407 to rewrite organic German documents as an encyclopedia entry, a Q&A dialogue or an educational-style…”

Abstention training with explicit negative examples used

“During training, we use training data samples where the correct answer is "I don't know".”

Add synthesized long-context conversations to the SFT mix used

“The long-context mix adds a 5 % share of the synthesised long-context conversations.”

Backfilling missing reasoning between tool calls used

“A teacher model writes reasoning that leads to the recorded tool call, which stays unchanged. The reasoning is kept only if a second LLM…”

Concatenate tool-use conversations into long-context synthesis tasks used

“Several such conversations are concatenated into long contexts that end with a request for an essay tying their topics together”

English-to-German translation for data augmentation used

“for Kolibri we relied on organic and rephrased German data alongside a carefully curated set of translated German data.”

Generate fully German reasoning-training samples used

“We instead generate samples in which prompt, reasoning, and answer are German; tool schemas, calls, and results stay unchanged.”

Generate unanswerable examples by masking context with helpful and adversarial agents used

“During training, the environment generates unanswerable samples from answerable ones by masking a sample’s context using a helpful agent…”

German reasoning prefill for synthetic teacher traces used

“A short German opener at the start of every teacher trace keeps the reasoning in German”

Merlin-Arthur synthetic negative-example generation procedure used

“we also train with our Merlin-Arthur procedure, developed in-house and explained in detail in our blog post, which solves data scarcity by…”

Multi-hop retrieval task synthesis using GLM-5.2 used

“with randomisations over a range of configurations of the provided tools, as well as alternative prompt formulations, and multi-hop task…”

Reference-blind German rewriting of math and science problems used

“We apply GLM-5.3 (Z.ai 2026c) to rewrite each problem in natural German without seeing the reference, so that the German problem does not…”

Regenerate training-data completions used

“For some open datasets, we regenerate the completions, because their answers fail our audits”

Regeneration of reasoning traces for agentic SFT data used

“For permissively available agentic SFT datasets, we regenerated the reasoning traces.”

Retain both reasoning and non-reasoning trajectory variants used

“These reasoning-backfilled conversations enter the data pool along with their non-reasoning counterparts.”

Search-dependence filtering for multi-hop questions used

“A hop is kept only if a search for its clue together with the previous answer finds the target document and a search for the clue alone…”

Section-grounded simulated requests for long-context training used

“Qwen3.8-27B (Qwen Team2026c) receives the document, an index of its sections and a prompt, and writes four different simulated user…”

Seeded planning of long-context question locations and answer shape used

“A seeded plan per document fixes which part of the document each question draws on (beginning, middle or several distant passages) and how…”

Spread long-context queries across document sections used

“Requests about the opening of a document alone would not train the model to use the rest of a long input, so the prompt asks for requests…”

Synthetic English pre-training rephrasing used

“Deduplicated English Common Crawl was rephrased with Gemma-4-26B-A4B following a Nemotron-CC-style recipe, using five prompt templates…”

Synthetic German web-data rephrasing used

“German synthetic rephrasings of web data were produced with Mistral-NeMo-12B.”

Synthetic German-language data for supervised fine-tuning used

“add synthetic German datasets to the SFT data pool”

Synthetic multi-hop retrieval-question generation used

“For each corpus, seed phrases taken mostly from document titles, supplemented by a hand-written list, are issued to the search tools…”

Synthetic-data training environments for domain-specific skills used

“we created training environments by generating high-quality specialized synthetic data capturing those required skills.”

Template-based synthetic data generation used

“using five prompt templates applied to each seed document: diverse QA pairs, distillation, knowledge list, knowledge extraction, and…”

Train value judgments using everyday requests without explicit ethical cues used

“The questions are phrased as everyday requests without words such as “ethical” or “moral”, as users rarely state that a request touches on…”

Train with full-context, Merlin-masked, and Morgana-masked question variants used

“Each question is posed three times, with the full context, with Merlin’s context and with Morgana’s, without marking which one it is”

User-simulated conversational continuation used

“We employ a user simulator to write follow-up turns in the style and intent of the original user, and have a teacher model answer them.”

Verifiable terminal-task generation by injecting repository bugs used

“For terminal tasks, GLM-5.3 (Z.ai 2026c) injects a small, realistic bug into a Python repository and writes a bug report from the user’s…”

data mixture & curriculum 29

Adaptive RL prompt and environment curriculum core

“The curriculum therefore focuses on prompts whose groups carry learning signal, and across environments, it scales down the weight of an…”

Preserve pre-training dataset proportions in mid-training default

“The optimised mixes and the production mix use the pre-training-delivered rule.”

Three-bucket training curriculum default

“we eventually decided to move to the simpler three-bucket approach (Section 3.2.2.5) to reduce the number of required hyperparameters.”

KL-regularized data-mixture optimization used

“subject to the repetition caps at a 5T budget and a KL penalty towards a reference mix, which keeps the solution near a hand-set mix.”

MergeMix used

“We use MergeMix to train on90B instead of 1057B tokens, i.e., 91 % less training compute, fine-tuning one specialist model per cluster.”

MergeMix expert-checkpoint merging used

“a merge of these expert checkpoints with the weights of a candidate mix approximates an 84B-token mid-training run on that mix.”

Bisection for shared reweighting-scale selection used

“The scale q is shared across all bands and is solved by bisection such that the reweighted component contributes its target budget”

Combined per-band sampling budgets used

“their sampling rates must be combined into a single per-band budget to avoid assigning the same documents independently to both shares”

Dirichlet sampling of data-mixture weights used

“Mixtures are drawn from a Dirichlet distribution centred on the natural token-count prior q∝ tokens, at 15 concentrations spaced…”

Empirical pre- and mid-training data-mix selection used

“Pre- and mid-training mixes were chosen empirically using mix-search proxy models trained with different mixes to fit a function which…”

FastText-quality gating of annotation scores used

“To condition the mix on fastText quality, we pair each score with either the identity or the indicator 1{fastText > τ}”

Four-epoch dataset repetition cap used

“The repetition cap of a dataset limits its draws across pre-training and mid-training combined to four epochs”

Gaussian-process surrogate optimization of annotation weights used

“We use these means in a Gaussian-process surrogate over the samples which then selects the optimum weights to minimise the mean BPB for…”

Interleave bulk-shuffled dataset indices used

“The loader index realises the schedule by interleaving the bulk-shuffled indices of the two mixes.”

Length-based reweighting of mid-training data used

“In the reweighted share, each document-length band is assigned a weight on a logistic curve over the band index b”

Linear interpolation from pre-training mix to mid-training mix used

“We mid-train Kolibri on 3.44T tokens of this mix and interpolate linearly from the pre-training mix to it over the first H = 100 B tokens.”

Log-linear benchmark models for optimizing mid-training domain weights used

“Following Olmix, we fit one log-linear law per benchmark to the expert-merge scores and solve for the weights that maximise the aggregate…”

Log-linear proxy model for predicting benchmark BPB from data mixture used

“To predict each benchmark’s BPB from a given data mix, we fit a log-linear law over the swarm of proxy models.”

Mid-rank normalization of annotation scores used

“rank(·) is the per-column mid-rank empirical cumulative distribution function.”

Optimize annotation mixture weights against proxy-model BPB used

“We run ablations with a 0.4B dense DCLM model trained to 1× Chinchilla on 8.2B randomly sampled tokens and record the mean BPB over all…”

Per-source repetition caps in data-mixture optimization used

“the caps κi = tokensi · r/R forbid a source from repeating more than r times at the target token budget R = 20T.”

Proxy-model mixture selection using fitted mixing laws used

“the pre-training data team trained thousands of small proxy models, each on a different mixture, and fitted a mixing law to their scores”

SFT sampling proportional to unique-token counts used

“each dataset sampled in proportion to its unique tokens (natural sampling, Section 2.3.3.3).”

Shard-proportional batch sampling by multivariate hypergeometric draw used

“To draw a batch, the collator splits the batch size across the shards with a single multivariate hypergeometric draw over their sizes…”

Shuffle buffer for conversation-level batch balance used

“A shuffle buffer keeps the samples of one conversation from dominating a batch.”

Training-data bias mitigation through reweighting and augmentation used

“We counteracted this by upsampling German data where downstream evaluations showed gains, and by adding translated and synthetically…”

Two-axis mid-training data mixing used

“The domain weights give each of the nine domains its share of the budget, and the in-domain composition splits the budget of a domain…”

Upweighting selected agentic and competitive-programming environments used

“We deliberately upweight several agentic environments as well as competitive programming, while keeping the total weight of the…”

Water-filling redistribution under repetition caps used

“Within each domain, a waterfill assigns every dataset its share under the sampling rule, pins a dataset whose share exceeds its cap at the…”

sequence packing 12

Cost-aware longest-first bin packing core

“We determine the number of bins with a greedy longest-first heuristic for bin packing, rounding up to a multiple of the number of ranks.”

Next-k-Fit core

“Conversations are assigned online with Next-k-Fit (Johnson 1974): each goes into the first of k open bins that still has room”

Bin packing of document chunks used

“by bin packing these chunks without padding tokens or unnecessary document cuts”

Exact-length sequence bin packing used

“fills each sequence of exactly the sequence length from a pool of 1024 pending chunks.”

Fixed-length conversation packing with document-masked attention used

“We follow common practice and pack several conversations into one sequence of fixed length with document-masked attention”

Oldest-chunk anchoring used

“The packer always anchors a sequence on the oldest chunk in its pool, which bounds how long any chunk can wait”

Per-rank buffering and ordering by attention cost used

“every rank buffers b packed sequences and orders them by their attention cost”

Random Sequential Document Buffer (RSDB) used

“Sequences are assembled online from a Random Sequential Document Buffer (RSDB) ... which spreads the chunks of a long document over many…”

RSDB document chunking used

“splits documents longer than the sequence length into chunks with the RSDB described below.”

Smallest-overfill fallback with truncation used

“If no subset fills the sequence exactly, the packer takes the smallest subset that overfills it and drops the tokens beyond the sequence…”

Subset-sum sequence filling used

“fills the remaining capacity with a subset-sum search over the rest.”

Uniform random buffered-document chunk selection used

“the worker draws a buffered document uniformly at random, emits the next contiguous chunk of at most one sequence length starting at that…”

tokenization 7

German morphology-aware tokenization core

“We also developed a tokenizer tailored to German word structure, so German text is processed efficiently without sacrificing English.”

UniBPE used

“it keeps the greedy bottom-up approach of BPE ... and scores merges with the Unigram loss objective”

Bilingual English-German tokenizer used

“We built a bilingual English-German tokenizer, trained and specialized on each model's pre-training dataset.”

Greedy bottom-up BPE merging used

“UniBPE keeps the greedy bottom-up approach of BPE”

Online document tokenization with end-of-sequence markers used

“tokenises each document online and appends the end-of-sequence token.”

UniBPE and standard-BPE vocabulary construction used

“standard BPE performs the next 7900 merges”

Unigram-objective merge selection used

“using the Unigram training objective for selecting which merge to add to the vocabulary.”

5 more mentioned or evaluated
Classifier ensembling for data-quality annotation evaluated

data filtering “We therefore experimented with ensembling approaches and compared this with a strong single-classifier baseline.”

FastText, embedding-MLP, and fine-tuned BERT classifier comparison evaluated

data filtering “We evaluate the classifiers on a held-out sample of 1M CC documents, sampled and judged independently.”

Training data mixing evaluated

data mixture & curriculum “Within a cluster, the datasets keep their proportions in the reference mix.”

Uniformly upsample German SFT data evaluated

data mixture & curriculum “we fine-tune the Kolibri base model with the ablation recipe of Table 54 on mixes that upsample all German data by the same factor”

Varying math-to-code proportions in a training data mix evaluated

data mixture & curriculum “math and code split the remaining 2/3 in the ratio α to 1−α”

Not used 1

Fixed-ratio environment sampling not used

data mixture & curriculum “This sampling restriction turned out to harm compute efficiency and did not improve empirical performance, so we dropped it.”

filed at the root 4

Matched-quantity context masking core

“The masks are ensured to hide similar amounts of context with some tolerance.”

Dataset-specific reference groups for reasoning-effort labeling used

“Therefore, we adopt dataset-specific reference groups.”

Reasoning-effort labeling used

“In SFT, we assigned reasoning effort labels based on observed dataset-specific reasoning length statistics and judged difficulty, while on…”

Seeded soft effort-label sampling around bucket boundaries used

“The draw is seeded per row, and its width keeps the label from switching at one exact token count, which the model could otherwise learn…”

supervised fine-tuning 11

German-language reasoning post-training core

“SFT cold-starts reasoning in German across all domains, so that the model reasons in German rather than only answering in it.”

Supervised fine-tuning used

“the first step is supervised fine-tuning to teach the model core reasoning ability and how to interact in a chat”

Assistant-only loss used

“SFT of Kolibri proceeds after the long-context training stage with cross-entropy loss on the assistant turns”

Mask loss on incorrect tool calls and terminal actions used

“During training, we masked out erroneous tool calls and terminal actions, so the model learned to recover from errors without learning the…”

Context-grounded abstention training used

“train the model to abstain when the provided context does not support an answer”

Fine-tuning a specialist model for each dataset cluster used

“datasets are grouped into clusters and a specialist is fine-tuned on each cluster”

Include English and German safety datasets in SFT used

“We therefore add both English and German open safety datasets to the SFT mix.”

Long-context post-training with document question answering used

“we curated datasets that contained long-context documents based on publicly available data (e.g., from German legislative documents) with…”

Mask loss on invalid or repeated tool calls used

“Finally, we mask the loss on clearly wrong tool calls: calls whose tool response reports an unknown tool or invalid arguments, and exact…”

Safety refusal training with appropriate partial answers used

“Kolibri was trained in SFT to decline harmful and illegal requests while still providing partial answers where appropriate.”

SFT refusal training for PII extraction attacks used

“we incorporated safety data in our SFT mix that provide refusals to PII extraction attacks.”

reinforcement learning algorithm 7

Group-relative advantage estimation core

“we compute the group-relative advantage of each segment as”

Difficulty-bucket and pass-rate curriculum core

“The curriculum sorts the prompts of each environment into three buckets by their estimated pass rate, namely hard prompts with an estimate…”

Merlin and Morgana masking algorithms core

“Arthur is the model that we train, i.e., the policy πθ, and Merlin and Morgana are masking algorithms.”

Dynamic sampling of informative rollout groups used

“we generally drop groups in which all sequences obtain the same environment reward.”

Large-scale reinforcement learning for long-horizon tasks used

“followed by large-scale reinforcement learning to train the model to reason across a diverse suite of long-horizon tasks.”

Mask re-rendered context during training used

“It then masks this context for the training steps to ensure that each generated token is only trained on once.”

Omitting advantage normalization used

“we skip the advantage normalisation originally used by GRPO”

reward modelling 16

Reinforcement Learning with Verifiable Rewards core

“All environments grade a sequence with a verifiable reward in [0, 1], except for the Hallucination Reduction environments, whose signed…”

Environment-specific output-format reward signals core

“Each of these requirements defines a format signal that influences training reward through the aggregation described next.”

Multiplicative aggregation of format signals core

“Format signals act multiplicatively on environment reward. They are only applied to positive rewards”

p-norm aggregation of format signals core

“The final reward Ri of environment i is computed by scaling the environment reward Renv i > 0 with a p-norm aggregation of format signals”

Role-centered reward baselines core

“Each of the three versions of a seed is rolled out 8 times, and each role is centred on its own mean”

Role-conditioned Merlin–Arthur rewards core

“A completion y receives a reward R(y) that rewards Arthur for doing what its role asks and penalises a wrong answer on the full context…”

Language-consistency reward used

“include a language consistency reward for both the reasoning trace and the answer”

Format reward conditioned on positive environment reward used

“split both reward types and only apply the format signals to sequences with positive environment reward.”

Hallucination mitigation through abstention and context-sensitive RL used

“To address hallucinations, we trained with abstention data. We also constructed RL environments which implement our Merlin-Arthur protocol.”

Killable, time-limited verifier process pool used

“Verifiers that execute untrusted or unbounded code on the rollout workers (Section 3.2.3) run in a killable process pool with a timeout…”

Merlin–Arthur correctness and abstention reward used

“The model is rewarded for the correct answer in the first two cases and for abstaining in the third”

Merlin–Arthur hallucination and abstention example construction used

“The same model generates completions on the full context and on both masked contexts (Deiseroth et al. 2026). We keep a seed when its…”

Multiplicative p-norm format reward used

“we switched from additive format signals to the multiplicative p-norm version of Equation (18).”

Reasoning-to-action verification used

“Each reasoning trace is verified against the tool call or answer that follows it.”

Regression-adjusted test-pass reward used

“the reward is the fraction of passing fail-to-pass tests floored by pass-to-pass tests regression.”

Test-case pass-fraction reward used

“The reward is the fraction of test cases passed.”

policy distillation 2

On-Policy Self-Distillation used

“OPSD aims to bootstrap the model’s capabilities via targeted hints for failures with respect to terminal code execution and tool calls.”

Reasoning-Prefill Distillation used

“Each opener is randomly picked per sample from a curated list per task domain, so that the model does not learn one fixed opening phrase.”

rollout & RL infrastructure 14

Asynchronous reinforcement learning core

“We trained in an asynchronous Reinforcement Learning setup, where vLLM inference ran in parallel to training.”

One rollout worker per non-controller node core

“Every node except the controller’s hosts a rollout worker that steps through the environments, as well as a sandbox service that executes…”

Routing sequences to the same inference replica core

“It ensures that all rollouts of a group of sequences and the subsequent steps of each rollout land on the same inference replica.”

Isolated resettable sandboxes for agent rollouts used

“To isolate it from the node and from other sandboxes, the instance runs in an unprivileged user namespace with its own mount, process and…”

Asynchronous batching of parallel retrieval queries used

“An asynchronous batcher coalesces the queries that parallel rollouts send at the same time into batched encodes on a small pool of…”

Capacity allocation balancing trainer and inference throughput used

“We therefore allocate nodes such that the trainer is barely the bottleneck during the later stages of training.”

Distributed rollout workers used

“We thus moved the threads into rollout workers on every node (Section 3.2.3).”

Maximum-staleness replay-buffer eviction used

“We bound the off-policy shift by evicting groups from the buffer once they exceed a maximum staleness”

Non-blocking weight synchronization to inference used

“Updated model weights were synchronized to the inference engine without waiting for ongoing generations to finish.”

Overlapped RDMA weight-transfer writes used

“This process overlaps the writes with the preparation of the next parameters and the gather of further parameters within the batch.”

Preserving the environment sampling mix used

“Rollout threads draw an environment per group rather than per unit of time, so the training batch follows the curriculum’s draws even…”

RDMA weight distribution through one replica per node used

“Additionally, only one inference replica per node receives the weights via RDMA.”

Replay-buffer shared-prefix deduplication used

“The replay buffer stores identical prefixes between two sequences only once to reduce storage overhead.”

Routed-Expert Replay used

“The inference engine records the experts that its MoE routers select for every generated token, allowing the trainer to replay these choices”

agentic post-training 23

Merlin–Arthur protocol core

“The hallucination reduction environments implement a Merlin–Arthur protocol (Deiseroth et al. 2026) on various RAG training datasets.”

Retrieval-augmented question-answering training core

“Two environments train this behaviour, and both are part of the RL mix”

Environment sampling proportional to sampling and curriculum weights core

“An environment is drawn with probability proportional to ωeVe, i.e., to its sampling weight ωe times the sum Ve =∑ b wb of its bucket…”

Prompt sampling by difficulty bucket and evaluation recency core

“Within an environment, the curriculum draws a bucket with probability proportional to its weight and takes the prompt at its head, the one…”

Randomization of RL environments and prompts core

“To make the model robust to various deployment scenarios, we randomise environments along three axes: the system prompt, the coding…”

Weighted environment sampling core

“Each environment carries a sampling weight, the rate at which the rollout threads draw samples.”

Agentic retrieval environments for RAG training used

“our retrieval environments contribute to an iterative engine that allows us to sharpen the model’s agentic RAG capabilities”

German-language RL training environments used

“we translated our English environment data into German, and include a language consistency format signal”

Harness randomization for agentic training used

“All tasks are heavily randomised over different harnesses, system prompts and languages to encourage generalisation.”

Long-context question-answering RL environment used

“Each question is answered eight times by a Kolibri checkpoint at low reasoning effort and eight times by GLM-5.2”

Morgana adversarial context hiding used

“The other, Morgana, hides parts of the context which decreases the probability of the correct answer, and we trained the model to abstain…”

Per-episode configuration randomization used

“To keep the model independent of any one setup, each episode draws its own configuration.”

Per-group randomization of system prompt, coding harness, and user request used

“The system prompt, the coding harness and the user request are randomised per group, so that the policy does not overfit to a single…”

Prompt typo injection used

“The user request has about 5 % of its lowercase words misspelled, sparing the words the verifier depends on.”

Randomized preservation of reasoning traces across turns used

“For each group, we sample a random reasoning effort level of Table 19 and additionally decide at random whether the model’s reasoning…”

Randomized reasoning-effort level per rollout group used

“For each group, we sample a random reasoning effort level of Table 19 and additionally decide at random whether the model’s reasoning…”

Randomized retrieval setup and task language used

“During training, we randomise application-side choices, such as the retrieval parameters, models and methods and the task language, so…”

Randomized training environments for real-world robustness used

“we additionally randomized the environments around concrete details (tools, configurations, harnesses).”

RL environments for political balance, copyright, and safety used

“We built an environment that rewards balanced answers on topics where source models show known political bias”

Self-hosted user simulator used

“We explored using external inference providers to host the user simulator, but moved to self-hosting to reduce cost and improve reliability.”

Separate abstention-only environment for unanswerable questions used

“A separate environment poses unanswerable questions with their full context and rewards only abstention.”

Tool-calling and terminal-environment post-training used

“During the RL stage, we trained on multiple tool-calling, software-engineering and terminal environments and randomized the harnesses to…”

Training environments that simulate real-world use cases used

“by simulating key aspects of real-world use-cases, such as harnesses, data distributions, and types of questions in our environments, we…”

mid-training & continual pretraining 1

Long-context extension via continued pretraining used

“We trained Kolibri on 768 B200 GPUs in three stages: 20T tokens of pre-training at a 16k sequence length over 21 days, 3.44T tokens of…”

4 more mentioned or evaluated
Long-context SFT data ablation evaluated

supervised fine-tuning “we fine-tune the proxy model with the ablation recipe with and without them”

Loss masking in supervised fine-tuning evaluated

supervised fine-tuning “The configurations differ in the data mix, the assignment of reasoning effort and the loss masking.”

Reasoning-effort assignment in SFT evaluated

supervised fine-tuning “The configurations differ in the data mix, the assignment of reasoning effort and the loss masking.”

Shortened supervised fine-tuning ablation recipe evaluated

supervised fine-tuning “Ablations train with a shorter recipe than the Kolibri SFT runs”

decoding strategy 1

Greedy decoding used

“For efficiency reasons, judge requests use greedy decoding without reasoning.”

reasoning control 5

Thinking mode core

“As summarised in Figure 26, the assigned label depends on the length of a reasoning trace relative to the other traces of the same dataset…”

System-prompt reasoning-effort conditioning default

“The chat template signals a chosen effort level to the model by adding a plain text section to the system prompt.”

Chat-template thinking-mode switch default

“Thinking is on by default. Turn it off for a request with `reasoning_effort: "none"` or `enable_thinking: false`”

Length-dependent completion-budget signal used

“further separates them via the length-dependent completion_budget format signal”

Soft probabilistic assignment of reasoning-effort labels used

“An exchange with L > 0 is placed on the unit interval by interpolating in log space through these two percentiles, which fall on the…”

serving parallelism 1

FP8 parameter quantization during transfer used

“Each parameter is quantised to FP8 on the assigned rank”

inference scheduling 1

Priority-based, topology-aware GPU job scheduling used

“We use Kueue for priority-based queueing and topology-aware scheduling to ensure jobs land within a single InfiniBand domain.”

agentic scaffolding 8

Kolibri 1 reasoning parser core

“The `kolibri1` reasoning parser reads the same switch as the chat template, so the response's `reasoning` and `content` are split correctly”

Multi-step retrieval with grounded answering and abstention core

“the model searches a corpus over several steps, answers from the retrieved text, and abstains when these sources do not contain the answer”

Egress allowlist and prompt hardening used

“in training, we harden our egress allowlist and system prompts to avoid model exploits.”

Search-efficiency constraints used

“The search must also be efficient: no duplicate calls, at most 30 % near-duplicate queries, at most three searches per required hop, and…”

Unprivileged-user sandboxes used

“We subsequently run sandboxes as unprivileged users”

Application-level output validation optional

“Conducting additional validations at the application level to ensure output quality and appropriateness, e.g. via Red-Teaming or…”

Hermes-style tool calling optional

“The serving command enables Hermes-style tool calling (--tool-call-parser kolibri1 --enable-auto-tool-choice).”

Independent agent review gate optional

“launches on a cluster can be gated by an independent review agent.”

filed at the root 3

Checkpoint abstraction core

“Savanna’s Checkpoint abstraction encapsulates the storage, retention, versioning, lineage and format conversion”

Cluster abstraction core

“Savanna’s Cluster abstraction encapsulates the differences in GPU type, workload manager, storage systems, and other cluster-specific…”

Composable typed pipeline functions core

“Composability comes from expressing manual steps as functions with typed inputs and outputs”

inference engine 3

One-GPU FP8 vLLM replicas core

“Each GPU hosts an independent vLLM (Kwon et al. 2023) replica of the full model in FP8 precision”

vLLM general-plugins entry point core

“vLLM finds the plugin through its `vllm.general_plugins` entry point”

vLLM plugin core

“A [vLLM](https://github.com/vllm-project/vllm) plugin that serves Aleph Alpha models.”

training framework 2

Declarative training-run configuration core

“A single TOML file describes a pre- or post-training run and its evaluation strategy end-to-end.”

Savanna core

“Savanna implements the Kolibri training recipe end-to-end and provides the framework for experimenting with it.”

infrastructure service 5

Allowlisted sandbox egress proxy used

“For environments that install dependencies, an egress proxy in the network namespace allows only package registries, GitHub and Hugging…”

Apptainer image conversion and caching used

“we convert each image on first use to a single-file Apptainer image (Kurtzer, Sochat, and Bauer 2017) and cache it on shared scratch.”

DeepEP v2 with ElasticBuffer used

“As expert parallel backend we use DeepEP v2 with the ElasticBuffer API”

Passive GPU health monitoring with NVIDIA DCGM used

“We do passive health monitoring via NVIDIA Data Center GPU Manager (DCGM).”

Shared read-only SquashFS root used

“All sandboxes of an image on a node share its SquashFS root, which the executor mounts read-only via squashfuse.”

filed at the root 12

Controlled proxy-scale ablation used

“We compare recipe choices through controlled proxy-scale ablations before validating at larger scale.”

Distributional correctness score used

“We score them with the distributional correctness score, alongside accuracy and a ternary score that rewards correct answers, penalises…”

Explicit abstention option used

“evaluate Kolibri Base on 7 English multiple-choice tasks that add an “I don’t know” (IDK) option to each question”

Flash-SFT proxy evaluation used

“Savanna allowed us to compose a flash-SFT stage onto each proxy model and evaluate it on the SFT benchmarks”

Grounding and hallucination evaluation used

“develop and track dedicated grounding and anti-hallucination measures”

IDF-weighted n-gram matching used

“The decon algorithm flags a document by matching Inverse Document Frequency (IDF)-weighted n-grams of the questions and answers, and a…”

Question-weighted IDF overlap thresholding used

“It scores the span as an IDF-weighted overlap ratio of the question and of the answer, weighted towards the question, and flags a document…”

Recall-calibrated decontamination threshold used

“We take the highest threshold that still finds them at a recall of 0.99.”

Sentence masking importance estimation used

“We approximate the importance of a sentence by measuring the difference in the probability the current model assigns to the ground truth…”

Separate question and answer n-gram indexes used

“Questions and answers are indexed separately as 5- and 3-grams over cl100k tokens.”

Standardized prompts and task settings used

“with identical prompts, few-shot configurations, and task settings”

Test-based injected-bug filtering used

“A bug is kept only if at least one test fails and at least half of the tests still pass, so that it is detectable but local.”

benchmark 4

Automatic checkpoint evaluation across capability benchmarks default

“Training checkpoints land roughly every hour and the pipeline automatically evaluates them on English and German knowledge, maths, code…”

Dedicated German evaluation suite used

“we built a dedicated German evaluation suite, combining hand-crafted and model-generated tasks, to better measure the impact of our German…”

Domain-specific evaluation suites used

“we first built domain-specific evaluation suites that mirror the skills, tool-calls, workflows, and edge cases required in key sectors”

Token-F1 grading for retrieval answers used

“graded retrieval answers by token F1 (Rajpurkar et al. 2016) (Appendix I.1.4) instead of the judge.”

evaluation harness 2

Savanna Benchmark abstraction core

“Savanna’s Benchmark abstraction encapsulates these together with the storage and lineage of the results”

Web-fetch removal from evaluation harnesses used

“removed web fetching from the evaluation harnesses”

judge 4

Frozen judge model used

“The frozen judge model (Section 3.2.3) is used only where the graded object is natural language, a conversation, or the model’s intent”

LLM judge for conversation classification used

“For datasets with open-ended answers, we judge the whole conversation, including reasoning traces, with gpt-oss-120b (OpenAI 2025) to…”

LLM judge for document-quality annotation used

“We first scored a fixed 1M-document English sample of our curated CC data with a judge, Qwen3-32B”

Schema-constrained decoding for judge outputs used

“For each prompt, the judge returns structured JSON, enforced via constrained decoding against a schema.”

averaging - mean@k; avg@k; pass@k; pass^k 2

Mean benchmark score used

“For each category and language, we report the mean score of its benchmarks. We report the mean over the categories of a language as…”

Pass@k used

“we sample multiple completions for each prompt from the initial SFT model and compute a pass@k plot”

human & real-world evaluation 4

Customer-proxy evaluation without customer data training core

“to improve on contextualised evaluations without ever training on customer data”

Customer-specific contextualized evaluation used

“in building environments which address these specifics directly, and thus reflect real-world use-cases”

Internal checkpoint preview and employee feedback used

“Colleagues reported failures the benchmarks did not show, and each round of feedback went into the environments and the training data of…”

Turn-level blind A/B testing used

“The chat interface also ran turn-level blind A/B tests against similar models, allowing us to gauge model capabilities during more…”

filed at the root 6

Re-render rollout context into training segments core

“If the resulting tokens no longer extend the current segment, e.g., because reasoning traces were removed or generated text re-tokenises…”

Human review before acting on model outputs used

“in which a person reviews the model's output before it is acted on rather than autonomous systems that act unreviewed.”

L-BFGS fitting with Huber loss and multi-start initialization used

“we fit over the swarm by limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) optimisation with a Huber loss and multi-start…”

Per-step adversarial context-mask recomputation used

“Merlin and Morgana recompute their masks from the current policy at every step.”

Power-law-with-floor loss extrapolation used

“We use a power law with floor ... in the cumulative-learning-rate coordinate”

Remove assistant-turn content when formatting tool calls used

“We empty the content of every assistant turn with tool_calls. Every step then holds only the reasoning, the tool calls and their results”

unfiled 42

Binary Total Variation trust-region mask core

“we replace standard PPO-style clipping with a B-TV mask.”

bounded agent action space enforced by safe-operation abstractions core

“the factory wraps storage, clusters and registries in abstractions that only permit safe operations”

Model Training as Code core

“We call this approach Model Training as Code (MTaC)”

recording training decisions in code comments and commit history core

“Provenance comes from code comments and commit history, which encode the learnings and decisions that led to main.”

nightly end-to-end small-model training and evaluation default

“Every night, our Model Factory trains and evaluates a small model end to end.”

Use the decon tool's default parameters default

“We keep AI2’s default parameters throughout.”

abstention instead of incorrect answering used

“it abstains instead of answering wrong on 44 % of AA-Omniscience items”

agent self-analysis of failures to improve instructions and harnesses used

“Agents also analyse their own failures and propose changes to their instructions and harnesses that improve accuracy and reduce time and…”

agent-driven monitoring and optimization of evaluation deployments used

“Agents monitor these deployments, trace inefficiencies and propose fixes.”

autonomous agent review of experiments and failed-run triage used

“agents autonomously review proposed experiments and triage failed runs.”

Avoid unsupervised use of foundation models in high-stakes environments used

“Avoid unsupervised use in high-stakes environments.”

B-TV token mask used

“the B-TV mask and the gradient-norm bound act on roughly 0.01 % of tokens per step.”

Completeness-and-soundness grounding score used

“The Merlin–Arthur protocol also gives us a hallucination measurement.”

Coordinate descent for iterative hyperparameter tuning used

“This allows us to use smaller grids for the search and employ coordinate descent to tune further hyperparameters iteratively.”

Discard the initial 20% of post-warmup observations before extrapolation fitting used

“For the method used, we discard the first 20 % of the post-warmup prefix before fitting.”

Document-level stream processing in the training data loader used

“unrolls them into individual documents.”

Fit effort thresholds per dataset-and-difficulty group used

“We therefore fit the thresholds per reference group g = (dataset, difficulty).”

Instrument transformer-block runtime with CUDA events used

“time each transformer block during a forward pass with CUDA events.”

Luxical embedding-based distilled judge classifiers used

“to balance inference speed over our corpus with quality, we opted to use the Luxical-based distilled judge classifiers when constructing…”

Memory-mapped BM25 lexical search index used

“Lexical search uses a memory-mapped bm25s index over the same chunks as vector search.”

Offline cross-GPU profiling and collective-kernel matching for straggler analysis used

“we take PyTorch profiles on all GPUs and match collective communication kernels across the profiles to figure out when each GPU joined…”

One-sided RDMA for training-batch transfer used

“We use one-sided RDMA for the transfer of training batches from the replay buffer to the trainer ranks.”

Online per-block compute-time comparison for straggler detection used

“We actively measure the compute time of each transformer block and compare them online.”

Online transformer-block compute-time monitoring used

“We actively measure the compute time of each transformer block and compare them online.”

Per-parallelism-dimension communication-straggler detection used

“We identify communication stragglers by comparing the collective communication arrival times and throughput across all parallelism…”

Per-sandbox size-capped tmpfs writable layer using OverlayFS used

“each sandbox adds its own size-capped tmpfs writable layer through OverlayFS, so no image is copied or extracted.”

Persistent Apptainer sandbox instance for an entire rollout used

“The sandbox itself is a long-running Apptainer instance that lives for the whole rollout.”

protected-branch training with reviewed pull-request changes used

“only changing it with carefully reviewed diffs applied via pull request.”

recording agent-discovered limitations as repository issues or code comments used

“They either report these limitations as issues in the factory’s repository, or mark them with # XXX comments in the code”

Training and validation sequence monitoring used

“we started recording all training and validation sequences on disk, gathering multiple terabytes per run.”

Training-loss extrapolation for hyperparameter selection used

“we use loss extrapolation ... to predict the learning-rate landscape at the target horizon and pick the best cell of a grid of Muon…”

trunk-based development with frequent small changes and targeted testing used

“we practise trunk-based development and land changes on main often and in small increments with targeted testing.”

Uniform checkpoint averaging used

“Every score is that of a merged checkpoint, the uniform average of the weights of the last k saved checkpoints”

Use maximum matching score over question-scaled data windows used

“we score the row in windows and keep the highest window matching score. Each window is 8 times as long as the question”

Validate tool-calling results in the calling system used

“provided the calling system validates the results.”

Value-alignment prompting around European democratic values used

“Open-source models answer the questions under a system prompt that centres human dignity, human rights, liberal democracy, the rule of law…”

Wrap expert dispatch and combine calls as PyTorch custom operators used

“We wrap buffer.dispatch() and buffer.combine() into PyTorch custom ops, so they seamlessly integrate into a fully PyTorch compiled…”

Application-layer logging of model outputs optional

“Logging of past model outputs on the application layer”

2 more mentioned or evaluated
Compare weighted specialist merges with proportionally mixed-data training evaluated

“For every weighting, we evaluate its merge, the specialists averaged with these shares, and a model trained for the same budget on data…”

Uniform checkpoint averaging across branch endpoint and pre-fork checkpoints evaluated

“We also evaluate a merged checkpoint obtained by uniformly averaging the branch endpoint with three pre-fork mid-training checkpoints…”

Not used 2

Omit LLM-judge quality filtering of chat replies not used

“We skip this filter, as it adds compute without a known benefit.”

Omit weight decay not used

“Each group maintains its own learning rate, and we do not use weight decay.”

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

SourceKindPublisherStatus
aleph-alpha.com/downloads/tech-report.pdftechnical reportAleph Alpharead
aleph-alpha.com/en/blog/kolibri-has-landed-a-sovereign-open-weight-modelofficial blogAleph Alpharead
huggingface.co/Aleph-Alpha/Kolibri-1model cardAleph Alpharead
github.com/Aleph-Alpha/aleph-alpha-inferencecode repoAleph Alpharead
↑