Aleph Alpha
Kolibri-1
What Kolibri-1’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 78B / 3.46B active
- Context
- 256K
- Attention
- sliding/global hybrid4:1 sliding-window (513) : global GQA, positional encoding only in SWA layers (NoPE global)
- Experts
- 384 experts: 1 shared + 6 routed active
- Input
- text
- Licence
- Apache 2.0
- Released
- 2026-10-03
Against the consensus recipe
Its documents state 9 of the 18 methods the field agrees on.
Data curation
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictionnot in its documents
- Grouped-query attentionnot in its documents
Optimization
- Muonused
- Expert parallelismnot in its documents
- Quantization-Aware Trainingused
Post-training
- Supervised fine-tuningused
- Asynchronous reinforcement learningcore
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modecore
- Multi-Token Predictionnot in its documents
- FP8not in its documents
Checked in its code
Its code shows 9 of the 19 architecture features checked — 3 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_kolibri1.py, configuration_kolibri1.py) and config.json at revision abfe85f673, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention code only | in its code | not stated | 48 query heads share 4 key/value heads via repeat_kv in every attention layer. show evidencenum_attention_heads = 48num_key_value_heads = 4 |
| Multi-head latent attention | not in its code | not stated | Attention uses standard separate q/k/v projections with no low-rank latent cache or kv_lora_rank, so MLA is not implemented. |
| Sliding-window attention | in its code | adopted | use_sliding_window is true and sliding_window=513, and layers tagged sliding_attention apply this window via… show evidenceuse_sliding_window is true and sliding_window=513, and layers tagged sliding_attention apply this window via create_sliding_window_causal_mask. use_sliding_window = truesliding_window = 513 |
| Interleaved sliding-window and global attention | in its code | adopted | layer_types interleaves four sliding_attention layers followed by one full_attention layer, repeating across all 50 layers. show evidencelayer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "sliding_atte… |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | not in its code | not stated | No indexer scoring module or top-k token selection mechanism exists in the attention code; config has no index_topk/index_n_heads keys. |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | All decoder layers use Kolibri1Attention (softmax attention with sliding or full mask); no recurrent/linear/state-space mixer layer type… show evidenceAll decoder layers use Kolibri1Attention (softmax attention with sliding or full mask); no recurrent/linear/state-space mixer layer type is present. |
| Gated DeltaNet layers | not in its code | not stated | No delta-rule recurrent update or decay gate is implemented anywhere in the modeling file. |
| Mamba-2 layers | not in its code | not stated | No SSM/Mamba-2 state-space block or related config keys (mamba_num_heads, ssm_state_size) appear in this model. |
| Learnable attention sink | not in its code | not stated | The attention softmax has no extra learned sink logit column; eager_attention_forward only computes query-key-value softmax. show evidence |
| Gated attention output | not in its code | not stated | Attention output goes directly from attn_output reshape to o_proj with no sigmoid gating multiplication applied. show evidence |
| QK normalization | in its code | adopted | Queries and keys are each passed through a per-head RMSNorm (q_norm/k_norm) before rotary embedding and attention. show evidence |
| Partial RoPE | not in its code | not stated | RoPE is computed over the full head_dim (rotate_half splits the whole dimension in half for rotation, not a partial subset), with no… show evidenceRoPE is computed over the full head_dim (rotate_half splits the whole dimension in half for rotation, not a partial subset), with no partial_rotary_factor or rope_dim config present. |
| YaRN RoPE scaling | code unclear | not stated | rope_parameters is read from config but its contents (rope_type, possible yarn scaling) are not shown in the given config.json, which only… show evidencerope_parameters is read from config but its contents (rope_type, possible yarn scaling) are not shown in the given config.json, which only lists rope_theta at top level. rope_theta = 10000.0 |
| Layers without positional encoding (NoPE) | in its code | adopted | Full-attention layers skip rotary position embedding entirely (NoPE) while sliding-window layers apply RoPE, per the is_sliding check. show evidencelayer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "sliding_atte… |
| Mixture of experts | in its code | adopted | Each token is routed via Kolibri1TopKRouter to 6 of 384 expert MLPs in Kolibri1SparseMoeBlock. show evidencenum_experts = 384num_experts_per_tok = 6 |
| Shared expert | in its code | adopted | A shared_experts MLP with nonzero intermediate size (512) is added to every token's output alongside routed experts. show evidenceshared_expert_intermediate_size = 512 |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | The router adds a learned e_score_correction_bias to logits only for top-k expert selection, not for the output weighting (which uses raw… show evidenceThe router adds a learned e_score_correction_bias to logits only for top-k expert selection, not for the output weighting (which uses raw sigmoid logits). |
| Multi-token prediction layers | not in its code | not stated | No multi-token-prediction modules or layer count config (num_nextn_predict_layers, mtp_num_layers) appear in the config or code. |
| SwiGLU feed-forward code only | in its code | not stated | Both the shared-expert MLP and the routed experts compute silu(gate(x)) * up(x) as hidden_act is silu. show evidencehidden_act = "silu" |
token mixer 3
softmax attention 2
global attention 1
“the other 10 use full attention without positional encoding”
sliding window attention 1
“restrict attention to a fixed-size window of preceding tokens, reducing both memory usage and computational cost”
hybrid layer stacking 1
“Kolibri uses full attention in every fifth block (10 of 50) and SWA in the other 40.”
channel mixer 9
mixture of experts 9
“Kolibri is the second model in the Kolibri family, an English–German MoE”
expert routing 1
“each MoE layer routes a token to 6 of its 384 routed experts”
expert load balancing 5
“Expert loads are balanced with Exact Quantile Balancing (EQB) and Load-Error Injection (LEI)”
“Expert loads are balanced with Exact Quantile Balancing (EQB) and Load-Error Injection (LEI)”
“To balance the workload across all expert parallel ranks, we rely on our MoE load balancing methodology as described in Section 2.1.5.”
“The expert-balancing bias was updated separately using a recentered global-quantile target update.”
“EQB (Neitemeier et al. 2026) computes this global quantile exactly for bfloat16 adjusted logits using two-pass radix selection”
fine-grained experts 1
“We used 384 smaller experts rather than fewer wide ones, as they performed better in our tests.”
positional encoding 2
“the full-attention blocks use No Positional Encoding (NoPE)”
“Qwen3.5 with static YaRN at factor 4 at every length”
normalization & residual 1
“Queries and keys are normalised per head with RMS normalisation (Henry et al. 2020; B. Zhang and Sennrich 2019) before RoPE”
context capacity 1
“evaluation beyond the training length needs no positional rescaling”
7 more mentioned or evaluated
fine-grained experts “We compare three expert configurations of about 80B total parameters”
normalization & residual “An adapted QK-Clip (Kimi Team 2025) for GQA with threshold 100 in place of QK normalisation also shows gradient-norm spikes.”
fine-grained experts “with width 2048 and 6 of 256 experts of hidden dimension 1024”
fine-grained experts “with width 2048 and 12 of 512 experts of the same hidden dimension”
expert routing “Random routed experts replaces each token’s selected experts with a uniform random draw and tests whether the routing decision matters”
“We compare residual widths of 640, 1024 and 1600 at a fixed depth of 50 layers.”
mixture of experts “silenced routed experts removes the routed output, keeping the shared expert, and tests whether the routed experts contribute at all.”
Not used 2
dense feed-forward network “Kolibri Origin uses dense FFNs in its first two blocks”
mixture of experts “We tried fixing this issue during SFT, where load balancing is disabled, through densifying layers 0 and 1 by concatenating the shared…”
language modelling objective 1
“pretrained the model with a causal next-token-prediction objective on a large and diverse document corpus”
auxiliary loss 3
“We approximate both masks by first ranking the sentences by how much masking each one individually changes the log-probability of a…”
“During training, we recompute the masks from the policy at every step”
“We add a squared log-ratio term to softly penalise policy drift”
optimization 54
filed at the root 3
“each parameter of a soup is the mean of its values in the two models.”
“Instead, we factorise the transfer across model size and token horizon”
“For the width transfer we use Maximal Update Parametrisation (µP)”
optimizer 8
“We optimise attention and expert weights with Muon”
“AdamW was used for the embedding matrix, one-dimensional backbone parameters, router weights, and language-model head.”
“it uses the optimiser groups of pre-training (Table 6), i.e., Muon (Jordan et al. 2024) for the attention and expert weights and Adam…”
“embedding weights and norm scales with Adam”
“We use decoupled weight decay for all optimiser groups except the embedding and norm scales.”
“we optimise the ratios between the learning rates of the different embedding, norm and head parameter groups as a factor of the Muon…”
“All other two-dimensional backbone parameters were optimized with Nesterov Muon using a learning rate of 0.001.”
“we adopt the spectral convention ... which keeps the Muon learning rate independent of model width”
learning-rate schedule 2
“we tune learning rates at the smallest ablation scale using η∈{η0/√2, η0, √2η0}”
“Kolibri is trained with a Warmup-Stable-Merge (WSM) recipe”
training precision 1
“with bfloat16 parameters, float32 reductions, and a maximum gradient norm of 1.0.”
quantization-aware training 2
“The trainer therefore uses FP8 quantisation-aware training (Jacob et al. 2017), using the same quantisation as the inference engine.”
“additionally used quantization aware training to enable efficient low-precision inference with vLLM.”
training stability 3
“We therefore directly bound the norm by scaling our objective”
“In addition to this per-token bound, we clip the global gradient norm.”
“added a per-token gradient-norm bound (Section 3.2.1.2) after noticing that gradients for individual tokens reached into the thousands.”
training parallelism 4
“As a remedy, we scale the batch size with the GPU count.”
“Dedicated NCCL process group for reduce-scatters, so they overlap with all-gathers instead of serialising behind them.”
“Keep weights unsharded after the backward in-between the two gradient accumulation steps, removes the need to all-gather weights starting…”
“Three reduce-scatter input buffers (about 800 MB each at this scale), to hide infrequent, non-deterministic communication latency that…”
training runtime 31
“Consensus comes from version control, where the main branch represents the team’s collective best understanding of how to train a model.”
“The Model Factory expresses these workflows as version-controlled code, which makes every experiment reproducible, composable, and…”
“We implemented the training pipeline as code, so that the learnings of our team landed in one versioned training recipe instead of…”
“Every proposed code change triggers a small end-to-end model run; training, evaluation, to find out if something broke within minutes.”
“We checkpoint every 250 training steps.”
“Every job pins versions of the code and of its artefacts (data, checkpoints, tokenisers), which we store immutably in a registry, so runs…”
“Checkpoint merging largely recovers this difference, with the four merged checkpoints indistinguishable within the standard error of the…”
“The trainer asynchronously saves a checkpoint consisting of model weights, optimiser states and learning-rate schedules at regular…”
“Our fault-tolerance setup combines frequent checkpointing with automatic recovery.”
“its full state, from the position in the index down to the buffer contents of every worker, is checkpointed”
“After every optimiser step, the trainer pushes updated weights to the inference replicas, as shown in Figure 37.”
“Our cluster orchestration automatically detects faulty nodes and swaps them out of the main training run for automatic reboot and/or…”
“We explicitly gather the runtimes on GPU/process 0 to get more detailed statistics than just the average runtime.”
“We then determine the base model weights as the average of the last 20 checkpoints of the long-context stage”
“we match the collective communication kernels between the profiles of all ranks compare their arrival time and observed throughput.”
“time each transformer block during a forward pass with CUDA events”
“Microbatch-aware collectives that defer the inter-replica all-reduce and FSDP2’s end-of-backward finalisation to the last microbatch.”
“we use double-buffering. This means that each training step evaluates the CUDA events of its preceding step.”
“It stages them in a double buffer, filling one half while the other is being sent.”
“Our fault-tolerance setup combines frequent checkpointing with automatic recovery.”
“The trainer uses full activation checkpointing.”
“each proxy training and evaluation task was queued independently such that it could fill the gaps between the larger jobs on the cluster.”
“each receiving replica converts the weights in place into its kernel format and broadcasts them over NVLink to the other replicas of its…”
“Savanna registers each intermediate checkpoint as it appears, merges it with the preceding checkpoints of the configured window, and…”
“Our main training run writes a PyTorch profile once every 15 min”
“For detecting complicated compute or communication straggler reasons, we use a profiling-based approach.”
“we re-use the same ElasticBuffer across all layers and recompute also the dispatch in the backward pass.”
“To reduce device memory consumption, we avoid checkpointing any activation inside the MoE feed forward network.”
“Our runs are GitHub Actions workflows, and reproducing one means checking out a commit.”
“To further reduce the memory consumption, we re-use the same ElasticBuffer across all layers”
“combine resulting checkpoints by souping, which averages the weights of runs trained on different mixes into one model”
data curation 140
filed at the root 1
“we complement German datasets with more than 2T German tokens that we curate from the web with our own pipeline and generate synthetically”
data sourcing 4
“Kolibri was trained on a filtered, bilingual (German/English) corpus combining curated web data, synthetic rephrasings and translations…”
“we assess Third-Party Datasets against the General Purpose AI Code of Practice commitments for direct crawling using all reasonably…”
“Long-context candidates were built by length-bucketing the mid-training pool and blending it with OCR'd PDFs.”
“We assess the source of the data contained in the dataset on possible indications of copyright infringements”
data filtering 41
“we remove from the training corpus every document that contains text from our evaluation suite.”
“A shared processing pipeline brings every sample into one canonical format, drops conversations that claim another model’s identity or…”
“we redact PII from all training sources. Each category is detected by a regular expression, checked by a validator that rejects false…”
“A document whose redacted bytes exceed 15 % of its length is dropped.”
“For German, we built upon our public work introduced in GermanWeb (T. F. Burns et al. 2026) by ensembling quality and grammar fastText…”
“The reference eval set contains all 226 benchmark tasks of our evaluation framework with all available splits (train, validation, and test)”
“On a hit, the inspection expands in both directions and keeps the evaluation instances that continue to match.”
“Kimi K2.6 rules on the disputed options without seeing the key.”
“we check candidate datasets before they enter the pool. Sample audits of several candidate datasets found many completions of mixed quality”
“drop every candidate whose content largely overlaps another”
“Every dataset of the pool passes the cleanup pass of Appendix D.1.1, exact deduplication across the whole pool (Appendix D.1.2), and PII…”
“enters the training set only if GLM-5.2 solved it at least once and Kolibri at most six times”
“To score the rest of the crawl, we distil each scalar judgement axis into a binary task (top quartile versus bottom quartile, discarding…”
“splitting each language into five equal-token-mass tiers.”
“Short matches must be perfect, which keeps common phrases from being flagged.”
“If we find indications that the entity crawling the data did not adhere to an appropriate opt-out, we exclude such datasets from our…”
“If we find indications for infringements on the data source level, we exclude the corresponding dataset from our model development.”
“We combined the Luxical classifiers with a binary fastText classifier trained to distinguish SFT-style data vs. random CC samples”
“The outputs were post-processed and filtered to remove LLM-generation artefacts and erroneous generations.”
“We build on the decon engine of Olmo 3 (Allen Institute for AI 2025; Ettinger et al. 2025) and match word n-grams with n = 5.”
“the average of two GermanWeb classifiers for German”
“For English web data, keyword-based filtering additionally removed sexually explicit content.”
“Language-agnostic heuristic filters and language identification route each document to an English or a German branch, which apply separate…”
“Finally, a heuristic gate checks that both reasoning and answer are German, and an LLM language healer fixes the errors it finds in the…”
“Web documents passed a heuristic filter pipeline inspired by the approaches described in Dolma 3, DCLM and RefinedWeb”
“fastText language identification routing documents into an English and a German branch with language-specific thresholds”
“we had to retune filtering parameters for the German language.”
“We therefore use FineWeb2’s German word-length settings, which allow a mean word length between 0 and 15 characters, instead of the 3 to…”
“Finally, a heuristic gate checks that both reasoning and answer are German, and an LLM language healer fixes the errors it finds in the…”
“We detect contamination with the algorithm of the decon tool (Allen Institute for AI 2025).”
“A pattern filter drops every conversation in which the assistant claims to be another model or to come from another lab, unless it also…”
“Combined classifier scores were rescaled to [0, 1] in the annotation pipeline, and each language corpus was sorted into 20…”
“We then reapply GLM-5.3 to solve the problem, and the answer is kept only if it matches the reference.”
“For SFT, we therefore remove English reasoning from exchanges in other languages”
“For the long-context mix, we removed synthetic long documents to avoid artificially inflating benchmarks such as RULER.”
“A sample whose trace remains malformed or switches language keeps its answer without reasoning”
“We combine five English per-document scores with weights on the probability simplex.”
“Another GLM-5.3 instance then fixes the bug in the Terminus 2 harness of Terminal-Bench (Merrill et al. 2026), and we keep the…”
“Our data pipeline filters against illegal, harmful, and pirated content and redacts personal data from data sources.”
“To prevent training on illegal, harmful, infringing or pirated content, we filtered our data based on a URL filter list.”
“We compute the IDF within each benchmark instead, so wording common to a benchmark counts for little.”
deduplication 10
“The Common Crawl corpus passed through exact/global (cross-dump) deduplication, MinHash fuzzy deduplication, and substring deduplication.”
“Exact deduplication removes byte-identical documents across all snapshots”
“first within each snapshot and then across all snapshots in chronological order, so the earliest copy remains”
“Documents are normalised as in RefinedWeb and CCNet (Wenzek et al.2020), with every digit replaced by a placeholder, and shingled into…”
“Exact deduplication removes every document whose UTF-8 text, without normalisation, has the same 128-bit xxHash as another document”
“The stage runs per language in 30 English and 6 German shards, to which the snapshots are dealt round-robin in chronological order”
“Fuzzy deduplication, which compares MinHash signatures of documents bucketed by locality-sensitive hashing, removes a further 25.9 % as…”
“keeps one occurrence of every repeated byte sequence of at least 500 bytes and cuts the others out of their documents.”
“Substring deduplication cuts repeated blocks of at least 500 bytes, such as navigation menus and page templates, out of the remaining…”
“every cluster of candidate pairs is verified on the Jaccard similarity of token 3-grams at a threshold of 0.80.”
synthetic data 30
“We produced synthetic data in English, using our deduplicated English CC data ... as seeds for rephrasing with Gemma-4-26B-A4B.”
“We use Mistral-Nemo-Instruct-2407 to rewrite organic German documents as an encyclopedia entry, a Q&A dialogue or an educational-style…”
“During training, we use training data samples where the correct answer is "I don't know".”
“The long-context mix adds a 5 % share of the synthesised long-context conversations.”
“A teacher model writes reasoning that leads to the recorded tool call, which stays unchanged. The reasoning is kept only if a second LLM…”
“Several such conversations are concatenated into long contexts that end with a request for an essay tying their topics together”
“for Kolibri we relied on organic and rephrased German data alongside a carefully curated set of translated German data.”
“We instead generate samples in which prompt, reasoning, and answer are German; tool schemas, calls, and results stay unchanged.”
“During training, the environment generates unanswerable samples from answerable ones by masking a sample’s context using a helpful agent…”
“A short German opener at the start of every teacher trace keeps the reasoning in German”
“we also train with our Merlin-Arthur procedure, developed in-house and explained in detail in our blog post, which solves data scarcity by…”
“with randomisations over a range of configurations of the provided tools, as well as alternative prompt formulations, and multi-hop task…”
“We apply GLM-5.3 (Z.ai 2026c) to rewrite each problem in natural German without seeing the reference, so that the German problem does not…”
“For some open datasets, we regenerate the completions, because their answers fail our audits”
“For permissively available agentic SFT datasets, we regenerated the reasoning traces.”
“These reasoning-backfilled conversations enter the data pool along with their non-reasoning counterparts.”
“A hop is kept only if a search for its clue together with the previous answer finds the target document and a search for the clue alone…”
“Qwen3.8-27B (Qwen Team2026c) receives the document, an index of its sections and a prompt, and writes four different simulated user…”
“A seeded plan per document fixes which part of the document each question draws on (beginning, middle or several distant passages) and how…”
“Requests about the opening of a document alone would not train the model to use the rest of a long input, so the prompt asks for requests…”
“Deduplicated English Common Crawl was rephrased with Gemma-4-26B-A4B following a Nemotron-CC-style recipe, using five prompt templates…”
“German synthetic rephrasings of web data were produced with Mistral-NeMo-12B.”
“add synthetic German datasets to the SFT data pool”
“For each corpus, seed phrases taken mostly from document titles, supplemented by a hand-written list, are issued to the search tools…”
“we created training environments by generating high-quality specialized synthetic data capturing those required skills.”
“using five prompt templates applied to each seed document: diverse QA pairs, distillation, knowledge list, knowledge extraction, and…”
“The questions are phrased as everyday requests without words such as “ethical” or “moral”, as users rarely state that a request touches on…”
“Each question is posed three times, with the full context, with Merlin’s context and with Morgana’s, without marking which one it is”
“We employ a user simulator to write follow-up turns in the style and intent of the original user, and have a teacher model answer them.”
“For terminal tasks, GLM-5.3 (Z.ai 2026c) injects a small, realistic bug into a Python repository and writes a bug report from the user’s…”
data mixture & curriculum 29
“The curriculum therefore focuses on prompts whose groups carry learning signal, and across environments, it scales down the weight of an…”
“The optimised mixes and the production mix use the pre-training-delivered rule.”
“we eventually decided to move to the simpler three-bucket approach (Section 3.2.2.5) to reduce the number of required hyperparameters.”
“subject to the repetition caps at a 5T budget and a KL penalty towards a reference mix, which keeps the solution near a hand-set mix.”
“We use MergeMix to train on90B instead of 1057B tokens, i.e., 91 % less training compute, fine-tuning one specialist model per cluster.”
“a merge of these expert checkpoints with the weights of a candidate mix approximates an 84B-token mid-training run on that mix.”
“The scale q is shared across all bands and is solved by bisection such that the reweighted component contributes its target budget”
“their sampling rates must be combined into a single per-band budget to avoid assigning the same documents independently to both shares”
“Mixtures are drawn from a Dirichlet distribution centred on the natural token-count prior q∝ tokens, at 15 concentrations spaced…”
“Pre- and mid-training mixes were chosen empirically using mix-search proxy models trained with different mixes to fit a function which…”
“To condition the mix on fastText quality, we pair each score with either the identity or the indicator 1{fastText > τ}”
“The repetition cap of a dataset limits its draws across pre-training and mid-training combined to four epochs”
“We use these means in a Gaussian-process surrogate over the samples which then selects the optimum weights to minimise the mean BPB for…”
“The loader index realises the schedule by interleaving the bulk-shuffled indices of the two mixes.”
“In the reweighted share, each document-length band is assigned a weight on a logistic curve over the band index b”
“We mid-train Kolibri on 3.44T tokens of this mix and interpolate linearly from the pre-training mix to it over the first H = 100 B tokens.”
“Following Olmix, we fit one log-linear law per benchmark to the expert-merge scores and solve for the weights that maximise the aggregate…”
“To predict each benchmark’s BPB from a given data mix, we fit a log-linear law over the swarm of proxy models.”
“rank(·) is the per-column mid-rank empirical cumulative distribution function.”
“We run ablations with a 0.4B dense DCLM model trained to 1× Chinchilla on 8.2B randomly sampled tokens and record the mean BPB over all…”
“the caps κi = tokensi · r/R forbid a source from repeating more than r times at the target token budget R = 20T.”
“the pre-training data team trained thousands of small proxy models, each on a different mixture, and fitted a mixing law to their scores”
“each dataset sampled in proportion to its unique tokens (natural sampling, Section 2.3.3.3).”
“To draw a batch, the collator splits the batch size across the shards with a single multivariate hypergeometric draw over their sizes…”
“A shuffle buffer keeps the samples of one conversation from dominating a batch.”
“We counteracted this by upsampling German data where downstream evaluations showed gains, and by adding translated and synthetically…”
“The domain weights give each of the nine domains its share of the budget, and the in-domain composition splits the budget of a domain…”
“We deliberately upweight several agentic environments as well as competitive programming, while keeping the total weight of the…”
“Within each domain, a waterfill assigns every dataset its share under the sampling rule, pins a dataset whose share exceeds its cap at the…”
sequence packing 12
“We determine the number of bins with a greedy longest-first heuristic for bin packing, rounding up to a multiple of the number of ranks.”
“Conversations are assigned online with Next-k-Fit (Johnson 1974): each goes into the first of k open bins that still has room”
“by bin packing these chunks without padding tokens or unnecessary document cuts”
“fills each sequence of exactly the sequence length from a pool of 1024 pending chunks.”
“We follow common practice and pack several conversations into one sequence of fixed length with document-masked attention”
“The packer always anchors a sequence on the oldest chunk in its pool, which bounds how long any chunk can wait”
“every rank buffers b packed sequences and orders them by their attention cost”
“Sequences are assembled online from a Random Sequential Document Buffer (RSDB) ... which spreads the chunks of a long document over many…”
“splits documents longer than the sequence length into chunks with the RSDB described below.”
“If no subset fills the sequence exactly, the packer takes the smallest subset that overfills it and drops the tokens beyond the sequence…”
“fills the remaining capacity with a subset-sum search over the rest.”
“the worker draws a buffered document uniformly at random, emits the next contiguous chunk of at most one sequence length starting at that…”
tokenization 7
“We also developed a tokenizer tailored to German word structure, so German text is processed efficiently without sacrificing English.”
“it keeps the greedy bottom-up approach of BPE ... and scores merges with the Unigram loss objective”
“We built a bilingual English-German tokenizer, trained and specialized on each model's pre-training dataset.”
“UniBPE keeps the greedy bottom-up approach of BPE”
“tokenises each document online and appends the end-of-sequence token.”
“standard BPE performs the next 7900 merges”
“using the Unigram training objective for selecting which merge to add to the vocabulary.”
5 more mentioned or evaluated
data filtering “We therefore experimented with ensembling approaches and compared this with a strong single-classifier baseline.”
data filtering “We evaluate the classifiers on a held-out sample of 1M CC documents, sampled and judged independently.”
data mixture & curriculum “Within a cluster, the datasets keep their proportions in the reference mix.”
data mixture & curriculum “we fine-tune the Kolibri base model with the ablation recipe of Table 54 on mixes that upsample all German data by the same factor”
data mixture & curriculum “math and code split the remaining 2/3 in the ratio α to 1−α”
Not used 1
data mixture & curriculum “This sampling restriction turned out to harm compute efficiency and did not improve empirical performance, so we dropped it.”
filed at the root 4
“The masks are ensured to hide similar amounts of context with some tolerance.”
“Therefore, we adopt dataset-specific reference groups.”
“In SFT, we assigned reasoning effort labels based on observed dataset-specific reasoning length statistics and judged difficulty, while on…”
“The draw is seeded per row, and its width keeps the label from switching at one exact token count, which the model could otherwise learn…”
supervised fine-tuning 11
“SFT cold-starts reasoning in German across all domains, so that the model reasons in German rather than only answering in it.”
“the first step is supervised fine-tuning to teach the model core reasoning ability and how to interact in a chat”
“SFT of Kolibri proceeds after the long-context training stage with cross-entropy loss on the assistant turns”
“During training, we masked out erroneous tool calls and terminal actions, so the model learned to recover from errors without learning the…”
“train the model to abstain when the provided context does not support an answer”
“datasets are grouped into clusters and a specialist is fine-tuned on each cluster”
“We therefore add both English and German open safety datasets to the SFT mix.”
“we curated datasets that contained long-context documents based on publicly available data (e.g., from German legislative documents) with…”
“Finally, we mask the loss on clearly wrong tool calls: calls whose tool response reports an unknown tool or invalid arguments, and exact…”
“Kolibri was trained in SFT to decline harmful and illegal requests while still providing partial answers where appropriate.”
“we incorporated safety data in our SFT mix that provide refusals to PII extraction attacks.”
reinforcement learning algorithm 7
“we compute the group-relative advantage of each segment as”
“The curriculum sorts the prompts of each environment into three buckets by their estimated pass rate, namely hard prompts with an estimate…”
“Arthur is the model that we train, i.e., the policy πθ, and Merlin and Morgana are masking algorithms.”
“we generally drop groups in which all sequences obtain the same environment reward.”
“followed by large-scale reinforcement learning to train the model to reason across a diverse suite of long-horizon tasks.”
“It then masks this context for the training steps to ensure that each generated token is only trained on once.”
“we skip the advantage normalisation originally used by GRPO”
reward modelling 16
“All environments grade a sequence with a verifiable reward in [0, 1], except for the Hallucination Reduction environments, whose signed…”
“Each of these requirements defines a format signal that influences training reward through the aggregation described next.”
“Format signals act multiplicatively on environment reward. They are only applied to positive rewards”
“The final reward Ri of environment i is computed by scaling the environment reward Renv i > 0 with a p-norm aggregation of format signals”
“Each of the three versions of a seed is rolled out 8 times, and each role is centred on its own mean”
“A completion y receives a reward R(y) that rewards Arthur for doing what its role asks and penalises a wrong answer on the full context…”
“include a language consistency reward for both the reasoning trace and the answer”
“split both reward types and only apply the format signals to sequences with positive environment reward.”
“To address hallucinations, we trained with abstention data. We also constructed RL environments which implement our Merlin-Arthur protocol.”
“Verifiers that execute untrusted or unbounded code on the rollout workers (Section 3.2.3) run in a killable process pool with a timeout…”
“The model is rewarded for the correct answer in the first two cases and for abstaining in the third”
“The same model generates completions on the full context and on both masked contexts (Deiseroth et al. 2026). We keep a seed when its…”
“we switched from additive format signals to the multiplicative p-norm version of Equation (18).”
“Each reasoning trace is verified against the tool call or answer that follows it.”
“the reward is the fraction of passing fail-to-pass tests floored by pass-to-pass tests regression.”
“The reward is the fraction of test cases passed.”
policy distillation 2
“OPSD aims to bootstrap the model’s capabilities via targeted hints for failures with respect to terminal code execution and tool calls.”
“Each opener is randomly picked per sample from a curated list per task domain, so that the model does not learn one fixed opening phrase.”
rollout & RL infrastructure 14
“We trained in an asynchronous Reinforcement Learning setup, where vLLM inference ran in parallel to training.”
“Every node except the controller’s hosts a rollout worker that steps through the environments, as well as a sandbox service that executes…”
“It ensures that all rollouts of a group of sequences and the subsequent steps of each rollout land on the same inference replica.”
“To isolate it from the node and from other sandboxes, the instance runs in an unprivileged user namespace with its own mount, process and…”
“An asynchronous batcher coalesces the queries that parallel rollouts send at the same time into batched encodes on a small pool of…”
“We therefore allocate nodes such that the trainer is barely the bottleneck during the later stages of training.”
“We thus moved the threads into rollout workers on every node (Section 3.2.3).”
“We bound the off-policy shift by evicting groups from the buffer once they exceed a maximum staleness”
“Updated model weights were synchronized to the inference engine without waiting for ongoing generations to finish.”
“This process overlaps the writes with the preparation of the next parameters and the gather of further parameters within the batch.”
“Rollout threads draw an environment per group rather than per unit of time, so the training batch follows the curriculum’s draws even…”
“Additionally, only one inference replica per node receives the weights via RDMA.”
“The replay buffer stores identical prefixes between two sequences only once to reduce storage overhead.”
“The inference engine records the experts that its MoE routers select for every generated token, allowing the trainer to replay these choices”
agentic post-training 23
“The hallucination reduction environments implement a Merlin–Arthur protocol (Deiseroth et al. 2026) on various RAG training datasets.”
“Two environments train this behaviour, and both are part of the RL mix”
“An environment is drawn with probability proportional to ωeVe, i.e., to its sampling weight ωe times the sum Ve =∑ b wb of its bucket…”
“Within an environment, the curriculum draws a bucket with probability proportional to its weight and takes the prompt at its head, the one…”
“To make the model robust to various deployment scenarios, we randomise environments along three axes: the system prompt, the coding…”
“Each environment carries a sampling weight, the rate at which the rollout threads draw samples.”
“our retrieval environments contribute to an iterative engine that allows us to sharpen the model’s agentic RAG capabilities”
“we translated our English environment data into German, and include a language consistency format signal”
“All tasks are heavily randomised over different harnesses, system prompts and languages to encourage generalisation.”
“Each question is answered eight times by a Kolibri checkpoint at low reasoning effort and eight times by GLM-5.2”
“The other, Morgana, hides parts of the context which decreases the probability of the correct answer, and we trained the model to abstain…”
“To keep the model independent of any one setup, each episode draws its own configuration.”
“The system prompt, the coding harness and the user request are randomised per group, so that the policy does not overfit to a single…”
“The user request has about 5 % of its lowercase words misspelled, sparing the words the verifier depends on.”
“For each group, we sample a random reasoning effort level of Table 19 and additionally decide at random whether the model’s reasoning…”
“For each group, we sample a random reasoning effort level of Table 19 and additionally decide at random whether the model’s reasoning…”
“During training, we randomise application-side choices, such as the retrieval parameters, models and methods and the task language, so…”
“we additionally randomized the environments around concrete details (tools, configurations, harnesses).”
“We built an environment that rewards balanced answers on topics where source models show known political bias”
“We explored using external inference providers to host the user simulator, but moved to self-hosting to reduce cost and improve reliability.”
“A separate environment poses unanswerable questions with their full context and rewards only abstention.”
“During the RL stage, we trained on multiple tool-calling, software-engineering and terminal environments and randomized the harnesses to…”
“by simulating key aspects of real-world use-cases, such as harnesses, data distributions, and types of questions in our environments, we…”
mid-training & continual pretraining 1
“We trained Kolibri on 768 B200 GPUs in three stages: 20T tokens of pre-training at a 16k sequence length over 21 days, 3.44T tokens of…”
4 more mentioned or evaluated
supervised fine-tuning “we fine-tune the proxy model with the ablation recipe with and without them”
supervised fine-tuning “The configurations differ in the data mix, the assignment of reasoning effort and the loss masking.”
supervised fine-tuning “The configurations differ in the data mix, the assignment of reasoning effort and the loss masking.”
supervised fine-tuning “Ablations train with a shorter recipe than the Kolibri SFT runs”
decoding strategy 1
“For efficiency reasons, judge requests use greedy decoding without reasoning.”
reasoning control 5
“As summarised in Figure 26, the assigned label depends on the length of a reasoning trace relative to the other traces of the same dataset…”
“The chat template signals a chosen effort level to the model by adding a plain text section to the system prompt.”
“Thinking is on by default. Turn it off for a request with `reasoning_effort: "none"` or `enable_thinking: false`”
“further separates them via the length-dependent completion_budget format signal”
“An exchange with L > 0 is placed on the unit interval by interpolating in log space through these two percentiles, which fall on the…”
serving parallelism 1
“Each parameter is quantised to FP8 on the assigned rank”
inference scheduling 1
“We use Kueue for priority-based queueing and topology-aware scheduling to ensure jobs land within a single InfiniBand domain.”
agentic scaffolding 8
“The `kolibri1` reasoning parser reads the same switch as the chat template, so the response's `reasoning` and `content` are split correctly”
“the model searches a corpus over several steps, answers from the retrieved text, and abstains when these sources do not contain the answer”
“in training, we harden our egress allowlist and system prompts to avoid model exploits.”
“The search must also be efficient: no duplicate calls, at most 30 % near-duplicate queries, at most three searches per required hop, and…”
“We subsequently run sandboxes as unprivileged users”
“Conducting additional validations at the application level to ensure output quality and appropriateness, e.g. via Red-Teaming or…”
“The serving command enables Hermes-style tool calling (--tool-call-parser kolibri1 --enable-auto-tool-choice).”
“launches on a cluster can be gated by an independent review agent.”
filed at the root 3
“Savanna’s Checkpoint abstraction encapsulates the storage, retention, versioning, lineage and format conversion”
“Savanna’s Cluster abstraction encapsulates the differences in GPU type, workload manager, storage systems, and other cluster-specific…”
“Composability comes from expressing manual steps as functions with typed inputs and outputs”
inference engine 3
“Each GPU hosts an independent vLLM (Kwon et al. 2023) replica of the full model in FP8 precision”
“vLLM finds the plugin through its `vllm.general_plugins` entry point”
“A [vLLM](https://github.com/vllm-project/vllm) plugin that serves Aleph Alpha models.”
training framework 2
“A single TOML file describes a pre- or post-training run and its evaluation strategy end-to-end.”
“Savanna implements the Kolibri training recipe end-to-end and provides the framework for experimenting with it.”
infrastructure service 5
“For environments that install dependencies, an egress proxy in the network namespace allows only package registries, GitHub and Hugging…”
“we convert each image on first use to a single-file Apptainer image (Kurtzer, Sochat, and Bauer 2017) and cache it on shared scratch.”
“As expert parallel backend we use DeepEP v2 with the ElasticBuffer API”
“We do passive health monitoring via NVIDIA Data Center GPU Manager (DCGM).”
“All sandboxes of an image on a node share its SquashFS root, which the executor mounts read-only via squashfuse.”
evaluation 28
filed at the root 12
“We compare recipe choices through controlled proxy-scale ablations before validating at larger scale.”
“We score them with the distributional correctness score, alongside accuracy and a ternary score that rewards correct answers, penalises…”
“evaluate Kolibri Base on 7 English multiple-choice tasks that add an “I don’t know” (IDK) option to each question”
“Savanna allowed us to compose a flash-SFT stage onto each proxy model and evaluate it on the SFT benchmarks”
“develop and track dedicated grounding and anti-hallucination measures”
“The decon algorithm flags a document by matching Inverse Document Frequency (IDF)-weighted n-grams of the questions and answers, and a…”
“It scores the span as an IDF-weighted overlap ratio of the question and of the answer, weighted towards the question, and flags a document…”
“We take the highest threshold that still finds them at a recall of 0.99.”
“We approximate the importance of a sentence by measuring the difference in the probability the current model assigns to the ground truth…”
“Questions and answers are indexed separately as 5- and 3-grams over cl100k tokens.”
“with identical prompts, few-shot configurations, and task settings”
“A bug is kept only if at least one test fails and at least half of the tests still pass, so that it is detectable but local.”
benchmark 4
“Training checkpoints land roughly every hour and the pipeline automatically evaluates them on English and German knowledge, maths, code…”
“we built a dedicated German evaluation suite, combining hand-crafted and model-generated tasks, to better measure the impact of our German…”
“we first built domain-specific evaluation suites that mirror the skills, tool-calls, workflows, and edge cases required in key sectors”
“graded retrieval answers by token F1 (Rajpurkar et al. 2016) (Appendix I.1.4) instead of the judge.”
evaluation harness 2
“Savanna’s Benchmark abstraction encapsulates these together with the storage and lineage of the results”
“removed web fetching from the evaluation harnesses”
judge 4
“The frozen judge model (Section 3.2.3) is used only where the graded object is natural language, a conversation, or the model’s intent”
“For datasets with open-ended answers, we judge the whole conversation, including reasoning traces, with gpt-oss-120b (OpenAI 2025) to…”
“We first scored a fixed 1M-document English sample of our curated CC data with a judge, Qwen3-32B”
“For each prompt, the judge returns structured JSON, enforced via constrained decoding against a schema.”
averaging - mean@k; avg@k; pass@k; pass^k 2
“For each category and language, we report the mean score of its benchmarks. We report the mean over the categories of a language as…”
“we sample multiple completions for each prompt from the initial SFT model and compute a pass@k plot”
human & real-world evaluation 4
“to improve on contextualised evaluations without ever training on customer data”
“in building environments which address these specifics directly, and thus reflect real-world use-cases”
“Colleagues reported failures the benchmarks did not show, and each round of feedback went into the environments and the training data of…”
“The chat interface also ran turn-level blind A/B tests against similar models, allowing us to gauge model capabilities during more…”
other 6
filed at the root 6
“If the resulting tokens no longer extend the current segment, e.g., because reasoning traces were removed or generated text re-tokenises…”
“in which a person reviews the model's output before it is acted on rather than autonomous systems that act unreviewed.”
“we fit over the swarm by limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) optimisation with a Huber loss and multi-start…”
“Merlin and Morgana recompute their masks from the current policy at every step.”
“We use a power law with floor ... in the cumulative-learning-rate coordinate”
“We empty the content of every assistant turn with tool_calls. Every step then holds only the reasoning, the tool calls and their results”
unfiled 42
“we replace standard PPO-style clipping with a B-TV mask.”
“the factory wraps storage, clusters and registries in abstractions that only permit safe operations”
“We call this approach Model Training as Code (MTaC)”
“Provenance comes from code comments and commit history, which encode the learnings and decisions that led to main.”
“Every night, our Model Factory trains and evaluates a small model end to end.”
“We keep AI2’s default parameters throughout.”
“it abstains instead of answering wrong on 44 % of AA-Omniscience items”
“Agents also analyse their own failures and propose changes to their instructions and harnesses that improve accuracy and reduce time and…”
“Agents monitor these deployments, trace inefficiencies and propose fixes.”
“agents autonomously review proposed experiments and triage failed runs.”
“Avoid unsupervised use in high-stakes environments.”
“the B-TV mask and the gradient-norm bound act on roughly 0.01 % of tokens per step.”
“The Merlin–Arthur protocol also gives us a hallucination measurement.”
“This allows us to use smaller grids for the search and employ coordinate descent to tune further hyperparameters iteratively.”
“For the method used, we discard the first 20 % of the post-warmup prefix before fitting.”
“unrolls them into individual documents.”
“We therefore fit the thresholds per reference group g = (dataset, difficulty).”
“time each transformer block during a forward pass with CUDA events.”
“to balance inference speed over our corpus with quality, we opted to use the Luxical-based distilled judge classifiers when constructing…”
“Lexical search uses a memory-mapped bm25s index over the same chunks as vector search.”
“we take PyTorch profiles on all GPUs and match collective communication kernels across the profiles to figure out when each GPU joined…”
“We use one-sided RDMA for the transfer of training batches from the replay buffer to the trainer ranks.”
“We actively measure the compute time of each transformer block and compare them online.”
“We actively measure the compute time of each transformer block and compare them online.”
“We identify communication stragglers by comparing the collective communication arrival times and throughput across all parallelism…”
“each sandbox adds its own size-capped tmpfs writable layer through OverlayFS, so no image is copied or extracted.”
“The sandbox itself is a long-running Apptainer instance that lives for the whole rollout.”
“only changing it with carefully reviewed diffs applied via pull request.”
“They either report these limitations as issues in the factory’s repository, or mark them with # XXX comments in the code”
“we started recording all training and validation sequences on disk, gathering multiple terabytes per run.”
“we use loss extrapolation ... to predict the learning-rate landscape at the target horizon and pick the best cell of a grid of Muon…”
“we practise trunk-based development and land changes on main often and in small increments with targeted testing.”
“Every score is that of a merged checkpoint, the uniform average of the weights of the last k saved checkpoints”
“we score the row in windows and keep the highest window matching score. Each window is 8 times as long as the question”
“provided the calling system validates the results.”
“Open-source models answer the questions under a system prompt that centres human dignity, human rights, liberal democracy, the rule of law…”
“We wrap buffer.dispatch() and buffer.combine() into PyTorch custom ops, so they seamlessly integrate into a fully PyTorch compiled…”
“Logging of past model outputs on the application layer”
2 more mentioned or evaluated
“For every weighting, we evaluate its merge, the specialists averaged with these shares, and a model trained for the same budget on data…”
“We also evaluate a merged checkpoint obtained by uniformly averaging the branch endpoint with three pre-fork mid-training checkpoints…”
Not used 2
“We skip this filter, as it adds compute without a known benefit.”
“Each group maintains its own learning rate, and we do not use weight decay.”
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| aleph-alpha.com/downloads/tech-report.pdf | technical report | Aleph Alpha | read |
| aleph-alpha.com/en/blog/kolibri-has-landed-a-sovereign-open-weight-model | official blog | Aleph Alpha | read |
| huggingface.co/Aleph-Alpha/Kolibri-1 | model card | Aleph Alpha | read |
| github.com/Aleph-Alpha/aleph-alpha-inference | code repo | Aleph Alpha | read |