During reinforcement learning for Mistral Large 4, one agent rollout ran for a day and a half. Mistral’s report: “The longest production-run rollout lasted about 37 hours and spanned 136 model updates.” Each update changed the weights the agent was sampling from, about one every 16 minutes (computed), so by the time the trajectory reached the trainer, its first tokens had been written by a model 136 updates old. At its peak the run had 26,000 rollouts in flight at once.
The policy gradient that RL training uses assumes that each sample was drawn from the policy being updated. Every measure that makes rollouts affordable breaks that assumption in its own way: not waiting for slow rollouts, generating on a separate inference engine in FP8, routing tokens to experts on a different machine, truncating the sampling distribution, re-encoding text between turns. The sections below take them one at a time, with what each lab does about it.
Background
The primer sets out where RL sits in a recipe, and S4 where the reward comes from. Here the reward is given, and the question is how the update uses it. The model being trained is the policy . For a prompt it writes a rollout , one token at a time. The policy gradient raises the log-probability of each token in proportion to how much better than expected the rollout turned out, its advantage :
The expectation is over rollouts drawn from itself: the on-policy assumption. Offline methods such as DPO (S4) sample nothing during training and are outside this article.
Rollouts are generated by an inference engine (vLLM or SGLang here), a serving program that batches thousands of sequences far better than a training framework can. As it samples each token, the engine records the probability it gave it, . The trainer (Megatron or FSDP) then runs the finished rollout through its own copy of the model in one forward pass. Three quantities are involved:
- : the trainer’s probability with its weights now.
- : the trainer’s probability with the weights of the version that generated the token. If training waits for every rollout, it equals at the first gradient step on a batch.
- : the engine’s recorded probability, from the weights but with different kernels, often a different number format and, in a mixture-of-experts model, its own choice of experts.
If the rollouts came from , the exact correction would multiply the ratios over every token of the rollout. Over 30,000 tokens that product is almost always near 0 or enormous, so every method here weights each token by its own importance weight
This corrects each token’s choice but not the prefix it was written after: the approximation behind TRPO and PPO (Schulman et al., 2015, 2017), exact only where . The first factor, staleness, grows with the number of policy updates between the weights that wrote a token and those training on it. The second exists even in a synchronous run, because two programs compute “the same” softmax differently. Z.ai’s reasoning RL corrects the two separately; Mistral, Xiaomi, Aleph Alpha and DeepSeek V3.2 use the single ratio , and Poolside a ratio to “the rollout policy at sampling time” without saying which program computes it.
On-policy GRPO
Group relative policy optimisation (GRPO, Shao et al., 2024) is the starting point for most labs here. It has no learned value function. For each prompt the policy writes a group of rollouts, and each rollout’s advantage comes from the group’s own rewards. In the original paper,
the same for every token of rollout ; the paper doesn’t say which standard deviation, and we use the population one. The update is PPO’s clipped surrogate with this advantage, averaged per sequence and then over the group, plus a penalty with coefficient on the KL divergence from a frozen reference model:
PPO takes several gradient steps on each batch, and from the second step on . For and , the min picks the clipped term , a constant, so the token’s gradient is zero; for the same happens below . At the ratio’s value is 1 but its gradient is , so DeepSeekMath, which made one update per batch, ran an on-policy policy gradient with a group baseline, reweighted per prompt by and per rollout by .
The group baseline suits a 0-or-1 verifier reward (S4). With 15 passes in a group of 16, each pass gets and the fail after subtracting the mean, and and after dividing by the standard deviation (computed). A group that all passes or all fails contributes nothing, and no critic network has to be trained.
Four of the nine labs that name or write down their RL objective call it GRPO: DeepSeek, Xiaomi, NVIDIA and Z.ai. Mistral, Poolside, Moonshot and Aleph Alpha are in the table at the end, and Qwen names GSPO for one speech-generation stage of Qwen3.5-Omni. Their equations differ. DeepSeek V3.2 writes the advantage as (its is our ), with no division by the standard deviation. Xiaomi’s “GRPO” in MiMo-V2.6 has neither the std nor PPO’s clip: it is a REINFORCE term with a stop-gradient importance weight and a token mask. For GLM-5’s reasoning RL, Z.ai’s “Training is performed entirely on-policy with a group size of 32 and a batch size of 32”, a setting it calls synchronous, and it still corrects the engine mismatch (IcePop, below): waiting for every rollout removes staleness and leaves the engine gap.
Variants of the baseline
Liu et al.’s critique of GRPO (“Dr. GRPO”, 2025) names two biases. Dividing each sequence’s loss by its length gives a short correct answer a larger per-token update and a long wrong answer a smaller penalty, so wrong answers can grow long. Dividing by the group’s standard deviation gives every group the same sum of squared advantages (), so a prompt that 1 rollout in 16 passes counts 4.1 times as much as with mean-centring alone, against 2.0 times for one that half pass (computed). Their fix removes both. DAPO (Yu et al., 2025) instead averages over all tokens in the batch, keeps the std, and drops all-pass and all-fail groups. Labs differ on these choices:
| Choice | Options | Who | Stated reason |
|---|---|---|---|
| Divide by the group’s std | yes / no / by the batch’s std | yes: Z.ai reasoning RL; no: Aleph Alpha, DeepSeek V3.2 (as written), Xiaomi (as written), Z.ai agentic RL; batch std (one scalar per step, so it rescales the step, not the prompts): Mistral | Aleph Alpha cites Dr. GRPO; Mistral gives none |
| Baseline | group mean / leave-one-out | mean: DeepSeek, Xiaomi, Z.ai, Mistral, Aleph Alpha, Moonshot (K2.5); leave-one-out weighted by rewarded tokens: Poolside | Poolside ablated against GRPO and GSPO, no numbers published |
| Groups with no signal | drop / keep some | drop: Xiaomi, Mistral, Aleph Alpha (which keeps all-pass groups whose format penalties differ) | Mistral: “the batch size counts only trajectories that contribute a gradient” |
| Loss aggregation | per token over the batch / per sequence / per prompt | token: Aleph Alpha, MiniMax-M1’s CISPO; sequence: DeepSeek V3.2, GLM-5 reasoning RL; per group in proportion to rollouts: Mistral; prompt: Xiaomi (text; its Eq. 1 prints a token mean) | Xiaomi: prompt-mean “prevents response length from growing too quickly” |
None reports an ablation of normalisation. Subtracting the group mean is the leave-one-out baseline (RLOO, Kool et al., 2019) scaled by , so the two differ only in Poolside’s weighting. Dropping a zero-signal group leaves the gradient’s direction unchanged; it changes the normaliser and, when dropped groups are re-sampled, the mix of prompt difficulties.
Stale rollouts
A synchronous step waits for its slowest rollout, and agentic rollouts are long-tailed: across Xiaomi’s 25 data sources, mean rollout duration varies 66-fold. NVIDIA writes that rollout time “is in turn dominated by a small fraction of straggler generations”. While they finish, the trainer and most inference GPUs sit idle.
Splitting one batch into several gradient steps, as PPO and DeepSeek V3.2 do, already makes every step after the first off-policy. The larger source is to stop waiting. In asynchronous RL, rollout workers generate continuously, the trainer takes finished rollouts as they arrive, and new weights are pushed to the engines while generation continues. Of the 16 labs, ten say how their rollouts are scheduled, and eight call their RL asynchronous: Aleph Alpha, DeepSeek (from V4.1), Mistral, NVIDIA, Qwen (in a model-card line), Thinking Machines, Xiaomi and Z.ai (for agentic RL only). Poolside runs the same way, with weight transfers that “run asynchronously”, without using the phrase for its RL. Moonshot calls its framework synchronous and gets stale data through partial rollouts: in Kimi K3 the generation phase “pauses as soon as a fraction λ∈(0,1) of trajectories completes”, and the rest resume under new weights. Moonshot runs rollout and training on the same GPUs in turn, so generation must stop for training (in DeepSeek V4.1, also co-located, “training preempts ongoing rollouts”); separate pools avoid the pause and pay in weight transfers.
In Aleph Alpha’s first setup, each weight update “waited a median of 221 s for running generations, and took another 217 s to load”. RDMA writes from every trainer rank (direct copies between GPU memories), updating in flight instead of draining, and a broadcast within each node “cut the weight update time down to about 17 s on comparable runs, which enabled weight updates after every training step”, 26 times less (computed). Mistral pushes weights “at each training step”, Poolside “every 2 optimizer steps”, Z.ai every updates (unstated), and Z.ai also resets the optimizer after each push, without a measurement.
The reports give staleness in different forms:
| Lab | Staleness | What it is |
|---|---|---|
| NVIDIA | 1 (2 for one Nemotron 3.5 Super teacher) | cap, in policy updates (“maximum trajectory age”) |
| Xiaomi | 4, plus a per-source limit | stated setting; units not given |
| Mistral | 6 | average, in steps |
| Poolside | 10 | cap, in optimizer steps; weights pushed every 2, so at most 5 engine versions (computed); never reached |
| Aleph Alpha | median 10.0; cap 42 | measured, from a group’s oldest start; groups over the cap evicted |
| Z.ai (agentic) | τ | drop a rollout if its oldest version is more than τ behind; value not printed |
| DeepSeek V4.1 | bounded | “we can bound the maximum off-policy ratio”; value not given |
Units differ, so does where the age is counted from, and so does how far a step moves the weights: 2,048 sequences at Aleph Alpha, about 7,000 usable trajectories at Mistral, 25,088 at Xiaomi (1,568 prompts times 16), under different optimizers and learning rates. Aleph Alpha explains why staleness rises when a run grows: it is roughly a rollout’s duration divided by the trainer’s step time. In its worked example a 60-minute code rollout with 2-minute steps is 30 steps stale, and doubling the GPUs at a fixed batch size makes it 60, past the cap of 42. “In some of our runs, code groups exceeded the staleness limit in this way after we scaled up the run.” Its remedy is to grow the batch with the GPU count.
staleness (weight updates during the rollout):01–23–56–1011–2021+slot idletrainer updatefirst batchover the cap, discardedstill running
Synchronous: 18 rollouts finished per hour; rollout slots idle 76% of the time; the trainer works 5% of it.
Asynchronous: 76 rollouts finished per hour; rollout slots idle 0%; the trainer waits 7.7 min for its first batch, then steps every 2 min.
Staleness: mean 4.2, maximum 30 (the 60-minute rollout; 60 / 2 = 30).
First batch: mean length 3.5 min, against 9.7 min over all rollouts.
A toy with invented lengths: a fixed log-normal sample, scaled so the longest equals the tail. The 12 asynchronous slots stand for a fleet that keeps the trainer busy, so after its first batch it steps every step time.
Asynchrony also biases which rollouts arrive first. DeepSeek V4.1: “shorter sequences tend to complete first and thus dominate the initial training batches”, especially early in training; it discards early short samples, and Xiaomi replays saved samples at start-up and after recovery. Restarting interrupted rollouts from scratch biases the same way (DeepSeek V4’s reason to resume them), and masking rollouts that hit a turn or time limit drops long ones by rule.
When new weights arrive mid-rollout, the engine holds a KV cache: the attention keys and values of every earlier token, kept so that each new token doesn’t recompute them (00). Labs do three different things with it.
- Reset: Poolside. “Weight broadcasts trigger an inference-side KV-cache reset, so the cache can never mix tokens encoded under different weight versions”, and an update waits for the current rollout step, so “from each rollout’s perspective the policy is piecewise-constant in time”.
- Rebuild: Xiaomi, which rebuilds the cache after each update and uses large batches to amortise the cost. Rebuilding, a prefill, is one forward pass over the whole context, up to 150K tokens here.
- Keep: Mistral, Aleph Alpha, DeepSeek V4.1 and, by implication, Kimi K3, whose paused rollouts keep their KV into the next iteration. Mistral: “its later tokens may attend to KV entries computed by earlier policies.” Aleph Alpha adds that its prefix cache also survives updates, so even a rollout started after an update can read an older policy’s keys and values for its prompt.
With a kept cache, the engine’s probability for a later token came from no single policy. The ratio still corrects that token, since is what the engine actually sampled with; what loses its meaning is , and with it the two-factor split. Poolside resets so that its ratio’s denominator is one weight version; Mistral and Aleph Alpha, which keep the cache, take theirs to , and DeepSeek V4.1 and Kimi K3 don’t say. No lab measures how much the mixing enlarges .
Corrections
Once , the update can use the per-token weight, which has high variance when a token is far likelier under one policy than the other, or limit it, which bounds the variance and adds bias. Labs use four kinds of rule.
Clip the weight, keep the token
CISPO (MiniMax-M1, 2025) came from a failure of PPO’s clip: rare tokens such as “However” and “Recheck”, which “often serve as “forks” in reasoning paths”, were clipped out after the first of 16 updates per batch, since a small change in a small probability is a large ratio. CISPO keeps every token and clips only its weight, treated as a constant:
with the stop-gradient; without the clip it is the per-token importance-weighted gradient. Mistral (CISPO-style REINFORCE loss) sets “with no lower bound”, on , so one weight covers both factors. Poolside, which pairs it with leave-one-out advantages, clips to ; its printed equation omits the stop-gradient. In expectation, clipping at gives the on-policy gradient with each token counted at probability instead of : tokens the trainer now favours count as less likely, and the mass they lose goes to no other token. Truncated importance sampling (TIS, Yao et al., 2025) caps the engine factor instead and keeps PPO’s clip on ; the name is used loosely, and DPPO’s “PG-TIS” is CISPO.
Mask outside a ratio band
IcePop (Ring-1T, 2025) multiplies each token by , with the engine factor alone: inside a band and 0 outside. Staleness is left to PPO’s clip on . Ring-1T used . In the same paper’s preliminary runs on Ring-mini-2.0, a 16.8B model, that band masked 1 to 2 tokens per thousand, and the narrower “immediately destabilizes training”. GLM-5 sets , exactly that band, for its synchronous reasoning RL and reports no such trouble; neither tested the other’s setup. Xiaomi masks on the single ratio , with one band for positive and one for negative advantages, “both initialized to [0.2, 5.0] and tuned independently at runtime based on policy entropy”. NVIDIA’s Nemotron 3 uses “masked importance sampling” with bounds outside our corpus, and Moonshot’s K2.5 (also outside it) masks on the log-ratio.
For agentic RL, where tracking across weight updates would need “an extensive history of model checkpoints”, Z.ai drops it. Its direct double-sided importance sampling (DDIS) weights each token by inside and masks it outside, on both sides; the bounds aren’t printed. That ratio is the importance weight itself. Z.ai credits the “controlled degree of off-policy bias” it accepts to reusing the rollout log-probabilities, but what that gives up is PPO’s separate anchor; the bias comes from the mask. A band, unlike PPO’s one-sided clip, also drops tokens whose update would shrink the mismatch, and since the advantage is one number per rollout, deleting tokens is not neutral even in expectation: a positive rollout moves probability toward the kept tokens, a negative one toward the deleted ones.
Mask on a probability difference
Aleph Alpha’s binary total-variation (B-TV) mask comes from DPPO (Qi et al., 2026), which argues that a ratio is the wrong measure of how far a token moved. A token going from to has ratio 100 and moves almost no probability mass; one going from 0.99 to 0.80 has ratio 0.81, inside any band, yet “removes 0.19 probability mass from the dominant token”. So the mask tests : with a positive advantage it masks a token whose trainer probability has risen more than 0.2 above the engine’s, with a negative one a token that has fallen more than 0.2 below it. Moves back toward are never blocked.
That leaves rare tokens: one going from 0.01 to 0.10 moves little mass, so B-TV keeps it at weight 10. The gradient on a token’s own logit is , close to for a rare token, and Aleph Alpha saw per-token gradients that “reached into the thousands”. It bounds that product at , capping the effective weight at . A plain cap such as CISPO’s, it argues, limits the weight “regardless of the training probability, and thus ignore[s] the softmax factor” . The mask and the bound together act on “roughly 0.01 % of tokens per step”.
Mask whole negative rollouts
DeepSeek V3.2’s off-policy sequence masking drops a rollout whose advantage is negative and whose mean over its tokens, an estimate of the drift, exceeds a threshold . The mask sits inside GRPO’s clipped objective, and DeepSeek’s “old” probability “denotes the sampling probability directly returned by the inference framework”, so PPO’s clip already bounds positive rollouts by . A negative token that the trainer now finds ten times likelier still passes at full weight. DeepSeek’s evidence is that the mask “improves stability in certain training scenarios that would otherwise exhibit instability”, with no numbers. Zeroing a negative rollout raises its advantage to 0, an optimistic bias toward the rollouts it masks.
Shown: “rare token gained” (engine probability μ = 0.01, trainer probability πθ = 0.10) with a negative advantage.
weight w = trainer probability πθ / engine probability μ = 0.10 / 0.010 = 10.00
for the Kolibri bound: 1 − πθ = 0.90, so the effective weight is capped at cmax / (1 − πθ) = 2 / 0.90 = 2.22
Three rules pass a weight of 10.0: per-token weight, B-TV, DeepSeek's clip and mask.
A toy for one token at θ = θold, so w = πθ / μ is pure engine mismatch. IcePop’s PPO clip on u is then inactive; DeepSeek V3.2’s clip is taken against the engine probability, so it applies on top of its mask. DeepSeek prints neither δ nor its clip ε: 0.5 and 0.2 are illustrative.
None of these is new to RL. Truncated weights are V-trace’s in IMPALA (Espeholt et al., 2018), built for the same lag between actors and learner; keeping a behaviour policy apart from a proximal anchor, as IcePop’s two ratios do, is the decoupled PPO objective of Hilton et al. (2022); DPPO’s total variation is the measure in TRPO’s original bound. What is new is the engine as a second source of .
The comparisons that exist are each on one model, and they disagree. IcePop’s paper reports beating TIS on a 16.8B model; the R3 paper (below) finds TIS helps plain GRPO but scores 1.6 to 1.7 points lower added to routing replay (two settings, one run each, best checkpoint); DPPO has two truncated variants “collapse prematurely”, one of them CISPO (its “PG-TIS”), in a 1.5B test it shares, with its authors, with the FP16 paper below.
KL to a reference
Classical RLHF penalises the KL divergence from the starting model to limit drift into reward hacking, estimated from sampled tokens. Schulman’s note on approximating from samples of compares three per-sample estimators, with : (unbiased, high variance), (biased, lower variance) and (unbiased, always positive). GRPO puts in the loss with .
DeepSeek V3.2 noticed that its tokens come from the engine, not from , and that plain fails “particularly when the sampled tokens have substantially lower probabilities under the current policy than the reference policy”. Its corrected estimate multiplies by , which in V3.2 is , so that “the gradient of this KL estimator becomes unbiased”. DeepSeek adds that “for certain domains, such as mathematics, applying a relatively weak KL penalty or even omitting it entirely can yield improved performance.”
Our derivation, checked numerically: used as a loss with samples from , has gradient , whose expectation is the gradient of , the divergence with its arguments swapped, while and DeepSeek’s weighted give the gradient of . Schulman’s note is about values, not gradients. With samples from a stale or mismatched , only the weighted form stays exact.
Mistral uses with , only “for a subset of data sources prone to over-optimization”. Z.ai removes the reference KL from reasoning RL “to accelerate RL improvement”, and Aleph Alpha uses none, as DAPO had argued. Aleph Alpha and Moonshot penalise drift from the rollout policy instead: Aleph Alpha adds with on every token, and Kimi K2.5 (outside our corpus; K3 follows its algorithm) has . Both are -shaped penalties on the importance weight: a trust region in penalty form, anchored to the rollout policy. K3 still streams a reference model’s weights during RL without saying what for. NVIDIA, Xiaomi and Poolside don’t state a reference KL either way.
Precision
Trainers compute in BF16, a 16-bit format; engines can serve in FP8, which halves the bytes each generated token reads. FP8 weights and an FP8 KV cache raised Aleph Alpha’s rollout throughput “by a factor of two to three in our runs”. The cost is a larger engine mismatch: now comes from rounded weights, from unrounded ones. Two labs disagree on what that does.
Aleph Alpha ran FP8 rollouts against an unrounded BF16 trainer and “observed RL training diverge in this configuration” (no curve given). Its fix is quantisation-aware training (QAT) on the trainer: the same FP8 rounding as the engine, computed in BF16. A residual remains: “the median training–inference mismatch |log rt| is roughly three times that of BF16 serving” (their is our ). Poolside tested the same arrangement before release and “observed no measurable hit to stability”, yet kept BF16 inference weights for the release runs of Laguna M.1 and XS.2, “as it was not obvious how this mismatch would play out over the many-day horizon of an RL run”; it later did RL “in FP8 precision” for Laguna S 2.1. The runs differ in length, hardware, correction rule and cache policy, so the disagreement is unresolved.
Of the five other labs that state rollout precision, four quantise both sides alike: Mistral (FP8 inference “to improve generation speed and match the trainer”), Xiaomi (experts in MXFP4, a 4-bit format, with the same rounding applied to the trainer’s weights after every update), Moonshot (“During RL, rollout and training share the same quantization scheme”) and DeepSeek V4 (FP4 QAT, and “native FP4 quantized weights” at rollout). Z.ai uses FP8 for rollouts without saying what the trainer does. An InfiXAI study (LMSYS blog, 2025) found that unified FP8 “effectively eliminates train–inference inconsistency caused by quantization error”.
Matched formats still leave some mismatch, because “none of the kernels are batch-invariant” (Aleph Alpha): sums taken in a different order round differently. NVIDIA (Nemotron 3.5 Super) and Aleph Alpha compute the output head in FP32; MiniMax-M1, which introduced this, reports the correlation of training and inference token probabilities rising from about 0.9 to about 0.99. DeepSeek V4 uses “batch-invariant and deterministic kernel libraries to ensure bitwise reproducibility across training and inference”. An outside paper (Qi et al., 2025) argues BF16 itself is the cause and that FP16, with three more mantissa bits, “effectively eliminates this mismatch”; no lab here reports trying it.
Expert routing
In a mixture-of-experts model a router picks a few experts per token per layer (C2). The pick is a top-k, and as Xiaomi puts it, “numerical differences between the engines can flip discrete expert selections”. The trainer then computes the token’s probability through experts the engine never used. On Qwen3-30B-A3B, the R3 paper measured “approximately 10% of the routers select different experts” between SGLang and Megatron, and “94% of tokens select a different expert in at least one” layer; the MoE model’s mismatch was 2.4 times the dense Qwen3-8B’s (computed from its values).
expertchosen (top-2)chosen differently from the engine
Routing replay records which experts the engine chose for every token and reuses them in the trainer’s forward pass. Five of the eight labs in the table below use it (all eight train MoE models; Moonshot, Poolside and Z.ai don’t state it): DeepSeek, which calls it Keep Routing and has used it “since DeepSeek-V3-0324”; Xiaomi, whose researchers proposed it as Rollout Routing Replay (R3); Mistral (“router replay”); Aleph Alpha (“Routed-Expert Replay”); and NVIDIA, in Nemotron 3.5 Super. The gate weights are recomputed from the trainer’s router logits, so the router keeps learning, at a rollout latency cost “below 3%” in the R3 paper. On one model, replay halved the measured mismatch and kept alive runs in which plain GRPO crashed between steps 60 and 120. DeepSeek V4.1 concatenates the routing of each segment for rollouts that span checkpoints. GSPO (Zheng et al., 2025) addresses a related flip inside the trainer, between and : it weights each rollout by the geometric mean of its token ratios and reports that this “has obviated the need” for replay there, on one model, without testing the engine gap.
Replay makes the trainer use the engine’s experts, but the router’s weights still change. On MiMo-V2.6-Pro, at decoder layer 9 (384 experts), Xiaomi measured that over the first 20 RL steps the coefficient of variation of expert load rose from 0.78 to 2.0 and the share of cold experts (load below a tenth of the mean) from 0.5% to 22%, about 84 of 384 (computed), while with the router frozen load stayed near its initial level. Restoring the router’s pre-RL values at step 20 brought load back “while benchmark performance remains unchanged”. Freezing the router is evidence about load and trainability, not scores (load balancing is C3); NVIDIA freezes it in one Nemotron 3.5 teacher. GLM-5’s sparse-attention indexer, another top-k (T3), had the same problem: a fast non-deterministic top-k kernel “caused drastic performance degradation during RL after only a few steps”, so Z.ai switched to the deterministic torch.topk and froze the indexer during RL.
Sampling and tokens
Rollouts are often sampled with top-p: the engine keeps only the most likely tokens whose probabilities add up to and renormalises over them. The trainer, computing over the full vocabulary, gives those tokens a smaller probability than did. The ratio is the nucleus’s mass, at least 0.97 at top-p 0.97, so no ratio rule ever fires on it, yet of up to 0.03 (computed) is about the size of the whole mismatch Aleph Alpha measures over its run. DeepSeek’s Keep Sampling Mask records the truncation set during sampling and applies it to in training, “ensuring both policies share identical action subspaces”; Xiaomi does the same. Aleph Alpha doesn’t truncate in RL (“We do not use top-p or top-k truncation”), at the price of training at top-p 1 while evaluating and serving at top-p 0.97 and top-k 128.
A multi-turn agent’s text goes back into the next prompt, and re-tokenising it can change the tokens: Mistral’s example is “the” followed by “re” becoming “there”. The trainer then scores tokens the engine never produced. Token-in-token-out keeps the sampled token IDs and passes only new text through the tokenizer: Z.ai finds it “critical for asynchronous RL training because it preserves exact action-level correspondence”, and Poolside and Xiaomi use it too, as does Moonshot’s K2.5. Mistral keeps re-tokenisation and handles the mismatch with FA4’s blockwise attention. Aleph Alpha declines token-in-token-out on purpose: “This choice is motivated by downstream usage during deployment, where the text might be re-encoded between turns.” Where the tokens no longer match, it splits the rollout and trains each piece as a separate segment, which “significantly increases the number of segments and thus cumulative tokens for some groups”.
Shorter rollouts
NVIDIA also shortens rollouts for some agentic tasks. PivotRL (Yi et al., 2026) runs GRPO on single turns from expert-trajectory states where sampled actions disagree in outcome; 71% of randomly chosen turns gave no signal. On software-engineering (SWE) tasks it reached 32.67%, a level end-to-end RL also reached, with about 4 times fewer rollout turns (computed from 133K and 542K); both were below the SFT baseline’s 37.40 on SWE-Bench Verified, while PivotRL beat SFT on three other agent benchmarks. Its states come from the expert, so the policy never trains on situations its own earlier turns create. Nemotron 3 Ultra uses it alone for the terminal and conversational tool-use teachers, and before a final end-to-end SWE-RL stage for the SWE teacher.
Scale
| Lab, run | Hardware | Per step | Total |
|---|---|---|---|
| Mistral Large 4 | 3,104 GPUs at peak, trainer to generator 1 : 2.5 | about 7,000 usable trajectories | 5.36M rollouts, 162.5B output tokens; up to 26,000 concurrent rollouts |
| MiMo-V2.6-Pro (Xiaomi) | “thousands of GPUs” | 25,088 sequences, 2.7B to 3.7B training tokens | 30 steps shown in 123 hours (Fig. 12); 2.6 million US dollars for RL post-training |
| Kolibri (Aleph Alpha) | 256 B300: 96 trainer, 144 inference, 16 judge | 2,048 sequences | 1,000 steps, 16,900 B300-equivalent GPU-hours; 25,000 sandboxes on average |
| Nemotron 3 Ultra (NVIDIA) | GB200 nodes | “global batch size of 8192, with each sample generating 16 rollouts” (prompts or trajectories not stated) | not stated |
Token counts differ in kind: Mistral’s 162.5B are generated output tokens, Xiaomi’s are training tokens including context (“roughly 110K–150K tokens per sequence”). Kimi K3 keeps each 1M-context RL experiment “within a few hundred GPUs”, and Thinking Machines scaled Inkling’s RL “to over 30M rollouts” in two long runs. Mistral, Aleph Alpha, Poolside, NVIDIA and Z.ai (agentic) keep separate rollout and training pools; Moonshot and DeepSeek V4.1 co-locate; Xiaomi doesn’t say.
What this can’t tell you
DeepSeek V4.1 writes “we refrain from introducing novel post-training algorithms” and credits “the scale, diversity, and verifiability of synthesized data and environments” with “essentially all of the observed gains”. The same report adds sample-level dispatch, concatenated routing replay, a bound on the off-policy ratio and a mask for stale tokens, so a release without “novel post-training algorithms” still carries much of this article’s machinery.
No lab publishes an ablation ranking one correction against another on its shipped model. Tried-and-dropped choices mostly go unrecorded; DeepSeek V4.1’s two rejected dispatch schemes (“severe oscillations in training metrics”, “stalled easily on long-tail samples”) are a rare exception. And the evidence is lopsided: Aleph Alpha’s report is the most detailed public account of most of these choices.
Comparison
| Mechanism | What it repairs | What it costs |
|---|---|---|
| Asynchronous RL, partial rollouts | GPUs idle behind long-tailed rollouts | staleness; short rollouts first; weight transfers or pauses |
| Staleness cap | very old samples | discarded rollouts, mostly long ones, so a bias against long solutions |
| Clip the weight, keep the token | variance from high-weight tokens | tokens the trainer now favours count as less likely |
| Mask outside a ratio band | tokens with large mismatch | drops tokens on both sides of the band; band to tune, and labs disagree on its width |
| Mask on probability difference | large moves in probability mass | rare tokens keep large weights, so it needs a gradient bound |
| Mask negative rollouts | off-policy failures | optimistic bias; one threshold |
| Reference KL | drift from the start | slows learning; estimator choice matters |
| Same quantisation both sides | FP8/FP4 engine mismatch | QAT in the trainer |
| Routing replay | flipped expert choices | recording and moving expert indices |
| Router freezing | router drift and expert overload | router stops adapting |
| Sampling-mask replay, or no truncation | top-p renormalisation | storing sets, or training off the serving distribution |
| Token-in-token-out | re-tokenisation drift | trains on token sequences deployment may not produce |
Where the labs sit
From the reports and cards; nothing here is checked against code. A blank means searched and not stated.
| Lab | Algorithm (lab’s word) | Baseline | Correction | Staleness | Routing | Rollout / trainer precision | KV across update | Reference KL |
|---|---|---|---|---|---|---|---|---|
| Aleph Alpha | unnamed (group-relative) | mean, no std; token mean | B-TV mask + gradient bound | median 10, cap 42 | replay | FP8 / FP8 QAT, FP32 head | keep | none ( instead) |
| DeepSeek | GRPO | mean (V3.2 formula) | PPO clip on + negative-rollout mask (V3.2); stale-token mask (V4.1) | bounded, value not given (V4.1) | Keep Routing since V3-0324 | FP4 / FP4 QAT (V4) | keep (V4.1) | corrected , weak or none for math (V3.2) |
| Mistral | CISPO-style REINFORCE | mean, batch std | weight clip at 3 | average 6 | replay | FP8 / FP8 | keep | , 0.005, some sources |
| Moonshot | K2.5’s algorithm | mean (K2.5) | log-ratio mask (K2.5) | partial rollout, no number | MXFP4 experts, MXFP8 activations, both sides | keep (implied) | none in K2.5’s objective; K3 streams a reference model, use not stated | |
| NVIDIA | asynchronous GRPO | masked importance sampling | 1 (2 for one teacher) | replay (3.5 Super) | FP32 head | |||
| Poolside | CISPO-style, leave-one-out | weighted leave-one-out | weight clip [0, 5] | cap 10 steps, never hit | BF16 + FP8 KV / BF16 (M.1, XS.2); FP8 (S 2.1) | reset | ||
| Xiaomi | GRPO | mean, prompt-mean loss | ratio-band mask, entropy-tuned | 4, per-source limit | R3, frozen router (V2.6) | MXFP4 / same MXFP4 rounding | rebuild | |
| Z.ai | GRPO + IcePop; DDIS for agents | mean and std (reasoning), mean (agentic) | IcePop band [0.5, 2]; DDIS band | on-policy (reasoning); τ (agentic) | deterministic indexer top-k | FP8 rollout / not stated | removed (reasoning) |
Qwen and Thinking Machines say their RL is asynchronous, and Qwen names GSPO for one stage of Qwen3.5-Omni; neither says more about these choices. Google, OpenAI, StepFun and Tencent publish nothing on them, and MiniMax’s corpus documents have nothing on M3’s RL (CISPO comes from its earlier M1 paper). Liquid AI’s LFM2 used offline preference optimisation; LFM3 adds online RL but describes none of these choices.
Easy to get wrong
- Treating as one rule. Xiaomi applies it to the trainer-over-engine weight; NVIDIA’s Nemotron 3.5 Super prints the same band for a ratio between two snapshots of the model in its distillation stage, a staleness ratio. And the IcePop paper found the that GLM-5 ships unstable, on its 16.8B Ring-mini-2.0.
- Comparing mismatch numbers. Mistral’s average of at most , Aleph Alpha’s median forward KL of per token with staleness included, and NVIDIA’s unquantified “more than an order of magnitude” reduction are different statistics on different models. At small mismatch the estimators roughly agree, but a mean is not a median.
- Taking router freezing as a quality gain. Xiaomi’s evidence is about load.
Check yourself
A group of 16 rollouts has 1 pass and 15 fails (rewards 1 and 0). What advantage does each get under the original GRPO and under DeepSeek V3.2’s written formula? How much more does this prompt count under the original?
The mean is . V3.2 subtracts it: the pass gets , each fail . The original divides by the standard deviation; with the population one (DeepSeekMath doesn’t say which), : and . Every advantage is scaled by , against 2.0 for a group that half passes, so this hard prompt gets about twice the relative weight of an even one. With the sample standard deviation the pass would get .
Poolside caps staleness at 10 optimizer steps and pushes weights to the engine every 2 steps. At most how many engine versions old can a rollout be, and why can’t you compare that with NVIDIA’s cap of 1?
At most 5 versions (10 / 2), since the engine only runs every second set of weights. NVIDIA counts policy updates, and the labs’ steps differ in batch size, learning rate and optimizer, so neither number says how far the weights moved.
A token has engine probability 0.01 and trainer probability 0.10, and its rollout has a negative advantage. What gradient coefficient does it get from Mistral’s clipped weight, GLM-5’s IcePop band, and Aleph Alpha’s B-TV mask with and without its gradient bound?
The weight is 10. Mistral’s is , so . GLM-5’s band excludes 10, so the token is masked: 0. The B-TV mask keeps it, since with a negative advantage it masks only tokens whose trainer probability fell more than 0.2 below the engine’s: . The gradient bound then caps the effective weight at , so about . The interactive figure reproduces these.
In the map
Methods
- Group Relative Policy Optimization 16 sources
- Asynchronous Group Relative Policy Optimization 3 sources
- Omitting advantage normalization 1 source
- Dropping zero-advantage rollout groups 2 sources
- CISPO-style REINFORCE objective 1 source
- CISPO with length-weighted leave-one-out group-relative advantages 1 source
- IcePop 2 sources
- Direct Double-sided Importance Sampling 1 source
- Binary Total Variation trust-region mask 1 source
- Off-Policy Sequence Masking 2 sources
- Importance-sampling-corrected unbiased KL estimate 2 sources
- Capping trajectory staleness 1 source
- Rollout Routing Replay 2 sources
- Keep Routing 1 source
- Freezing the MoE router during reinforcement learning 1 source
- Keep Sampling Mask 2 sources
- Token-in-Token-out (TITO) 2 sources
- Partial rollout 3 sources
- Asynchronous reinforcement learning 14 sources
- PivotRL 1 source
Taxonomy nodes
- reinforcement learning algorithm 75 methods
- rollout & RL infrastructure 98 methods
Further reading
- Shao et al., DeepSeekMath, arXiv:2402.03300, and Schulman et al., PPO, arXiv:1707.06347: GRPO and the clipped surrogate.
- Espeholt et al., IMPALA, arXiv:1802.01561, and Hilton et al., Batch size-invariance for policy optimization (decoupled PPO), arXiv:2110.00641: V-trace’s truncated weights for lagging actors, and separating the behaviour policy from the proximal anchor.
- MiniMax-M1, arXiv:2506.13585: CISPO and the FP32 head.
- Liu et al., Dr. GRPO, arXiv:2503.20783, and Yu et al., DAPO, arXiv:2503.14476: the length and difficulty biases, clip-higher and dynamic sampling, with ablation tables.
- Qi et al., Rethinking the Trust Region (DPPO), arXiv:2602.04879: the case against ratio clipping and the B-TV and B-KL masks.
- Ring-1T, arXiv:2510.18855: IcePop, its compounding-mismatch argument and its band sensitivity test.
- Ma et al., Rollout Routing Replay, arXiv:2510.11370, and Zheng et al., GSPO, arXiv:2507.18071: replay’s design and measurements, and sequence-level ratios with replay between policy versions.
- Qi et al., FP16, arXiv:2510.26788, and the InfiXAI FP8 post (lmsys.org/blog/2025-11-25-fp8-rl/): where numeric mismatch comes from.
- Schulman, Approximating KL Divergence (joschu.net/blog/kl-approx.html); Kimi k1.5, arXiv:2501.12599 (partial rollouts); Yi et al., PivotRL, arXiv:2603.21383.
- The labs’ own accounts: Aleph Alpha’s Kolibri report §3.2 and Appendix I (the most detailed), Mistral Large 4 report §4.3 to §4.4, MiMo-V2.6 report §4 to §6, Laguna report §4.4, GLM-5 §3.2 and §4.1, DeepSeek V3.2 §3.1 and V4.1 §5.
