Open Models Almanac

Training stages · S5

RL at scale: GRPO and the gap between sampler and trainer

Asynchrony, low-precision rollouts and a separate inference engine make RL affordable, and each makes the samples come from a slightly different policy than the one being trained

Illustration of a long paper tape running out of a small ink-blue printing press on the left; further along, the tape passes under a row of teal and coral stamps, each a slightly different shape, and a muted-gold ruler lies across the tape where it ends.

During reinforcement learning for Mistral Large 4, one agent rollout ran for a day and a half. Mistral’s report: “The longest production-run rollout lasted about 37 hours and spanned 136 model updates.” Each update changed the weights the agent was sampling from, about one every 16 minutes (computed), so by the time the trajectory reached the trainer, its first tokens had been written by a model 136 updates old. At its peak the run had 26,000 rollouts in flight at once.

The policy gradient that RL training uses assumes that each sample was drawn from the policy being updated. Every measure that makes rollouts affordable breaks that assumption in its own way: not waiting for slow rollouts, generating on a separate inference engine in FP8, routing tokens to experts on a different machine, truncating the sampling distribution, re-encoding text between turns. The sections below take them one at a time, with what each lab does about it.

Background

The primer sets out where RL sits in a recipe, and S4 where the reward comes from. Here the reward rr is given, and the question is how the update uses it. The model being trained is the policy πθ\pi_\theta. For a prompt xx it writes a rollout y=(y1,…,yn)y = (y_1, \dots, y_n), one token at a time. The policy gradient raises the log-probability of each token in proportion to how much better than expected the rollout turned out, its advantage AA:

∇θJ=Ey∼πθ[∑tA ∇θlog⁡πθ(yt∣x,y<t)].\nabla_\theta J = \mathbb{E}_{y \sim \pi_\theta}\Big[\sum_t A \,\nabla_\theta \log \pi_\theta(y_t \mid x, y_{<t})\Big].

The expectation is over rollouts drawn from πθ\pi_\theta itself: the on-policy assumption. Offline methods such as DPO (S4) sample nothing during training and are outside this article.

Rollouts are generated by an inference engine (vLLM or SGLang here), a serving program that batches thousands of sequences far better than a training framework can. As it samples each token, the engine records the probability it gave it, μ(yt)\mu(y_t). The trainer (Megatron or FSDP) then runs the finished rollout through its own copy of the model in one forward pass. Three quantities are involved:

  • πθ\pi_\theta: the trainer’s probability with its weights now.
  • πold\pi_{\text{old}}: the trainer’s probability with the weights of the version that generated the token. If training waits for every rollout, it equals πθ\pi_\theta at the first gradient step on a batch.
  • μ\mu: the engine’s recorded probability, from the πold\pi_{\text{old}} weights but with different kernels, often a different number format and, in a mixture-of-experts model, its own choice of experts.

If the rollouts came from μ\mu, the exact correction would multiply the ratios πθ(yt)/μ(yt)\pi_\theta(y_t)/\mu(y_t) over every token of the rollout. Over 30,000 tokens that product is almost always near 0 or enormous, so every method here weights each token by its own importance weight

wt=πθ(yt)μ(yt)=πθ(yt)πold(yt)⏟ut: staleness⋅πold(yt)μ(yt)⏟ρt: engine mismatch.w_t = \frac{\pi_\theta(y_t)}{\mu(y_t)} = \underbrace{\frac{\pi_\theta(y_t)}{\pi_{\text{old}}(y_t)}}_{u_t:\ \text{staleness}} \cdot \underbrace{\frac{\pi_{\text{old}}(y_t)}{\mu(y_t)}}_{\rho_t:\ \text{engine mismatch}}.

This corrects each token’s choice but not the prefix it was written after: the approximation behind TRPO and PPO (Schulman et al., 2015, 2017), exact only where πθ=μ\pi_\theta = \mu. The first factor, staleness, grows with the number of policy updates between the weights that wrote a token and those training on it. The second exists even in a synchronous run, because two programs compute “the same” softmax differently. Z.ai’s reasoning RL corrects the two separately; Mistral, Xiaomi, Aleph Alpha and DeepSeek V3.2 use the single ratio wtw_t, and Poolside a ratio to “the rollout policy at sampling time” without saying which program computes it.

Figure 1.One rollout, several authors. The trainer pushes new weights while a long rollout is still being written, so its segments come from different versions, and if the engine keeps its KV cache, later segments read entries that earlier versions wrote. The engine records the probability it sampled each token with; the trainer recomputes its own. For a token written by one version, the weight splits into a staleness and an engine-mismatch factor. Mistral's longest rollout spanned 136 updates.

On-policy GRPO

Group relative policy optimisation (GRPO, Shao et al., 2024) is the starting point for most labs here. It has no learned value function. For each prompt the policy writes a group of GG rollouts, and each rollout’s advantage comes from the group’s own rewards. In the original paper,

Ai=ri−mean⁡(r1,…,rG)std⁡(r1,…,rG),A_i = \frac{r_i - \operatorname{mean}(r_1, \dots, r_G)}{\operatorname{std}(r_1, \dots, r_G)},

the same for every token of rollout ii; the paper doesn’t say which standard deviation, and we use the population one. The update is PPO’s clipped surrogate with this advantage, averaged per sequence and then over the group, plus a penalty with coefficient β\beta on the KL divergence from a frozen reference model:

J=1G∑i=1G1∣yi∣∑t[min⁡(ui,tAi, clip⁡(ui,t,1−ε,1+ε) Ai)−β KLt],ui,t=πθ(yi,t)πold(yi,t).J = \frac{1}{G}\sum_{i=1}^{G} \frac{1}{|y_i|}\sum_{t} \Big[\min\big(u_{i,t} A_i,\ \operatorname{clip}(u_{i,t}, 1-\varepsilon, 1+\varepsilon)\, A_i\big) - \beta\, \mathrm{KL}_t\Big], \qquad u_{i,t} = \frac{\pi_\theta(y_{i,t})}{\pi_{\text{old}}(y_{i,t})}.

PPO takes several gradient steps on each batch, and from the second step on u≠1u \ne 1. For A>0A > 0 and u>1+εu > 1 + \varepsilon, the min picks the clipped term (1+ε)A(1+\varepsilon)A, a constant, so the token’s gradient is zero; for A<0A < 0 the same happens below 1−ε1 - \varepsilon. At u=1u = 1 the ratio’s value is 1 but its gradient is ∇log⁡πθ\nabla \log \pi_\theta, so DeepSeekMath, which made one update per batch, ran an on-policy policy gradient with a group baseline, reweighted per prompt by 1/std1/\text{std} and per rollout by 1/length1/\text{length}.

The group baseline suits a 0-or-1 verifier reward (S4). With 15 passes in a group of 16, each pass gets +0.06+0.06 and the fail −0.94-0.94 after subtracting the mean, and +0.26+0.26 and −3.87-3.87 after dividing by the standard deviation (computed). A group that all passes or all fails contributes nothing, and no critic network has to be trained.

Four of the nine labs that name or write down their RL objective call it GRPO: DeepSeek, Xiaomi, NVIDIA and Z.ai. Mistral, Poolside, Moonshot and Aleph Alpha are in the table at the end, and Qwen names GSPO for one speech-generation stage of Qwen3.5-Omni. Their equations differ. DeepSeek V3.2 writes the advantage as Ri−mean⁡(R)R_i - \operatorname{mean}(\mathbf{R}) (its RR is our rr), with no division by the standard deviation. Xiaomi’s “GRPO” in MiMo-V2.6 has neither the std nor PPO’s clip: it is a REINFORCE term with a stop-gradient importance weight and a token mask. For GLM-5’s reasoning RL, Z.ai’s “Training is performed entirely on-policy with a group size of 32 and a batch size of 32”, a setting it calls synchronous, and it still corrects the engine mismatch (IcePop, below): waiting for every rollout removes staleness and leaves the engine gap.

Variants of the baseline

Liu et al.’s critique of GRPO (“Dr. GRPO”, 2025) names two biases. Dividing each sequence’s loss by its length gives a short correct answer a larger per-token update and a long wrong answer a smaller penalty, so wrong answers can grow long. Dividing by the group’s standard deviation gives every group the same sum of squared advantages (GG), so a prompt that 1 rollout in 16 passes counts 4.1 times as much as with mean-centring alone, against 2.0 times for one that half pass (computed). Their fix removes both. DAPO (Yu et al., 2025) instead averages over all tokens in the batch, keeps the std, and drops all-pass and all-fail groups. Labs differ on these choices:

ChoiceOptionsWhoStated reason
Divide by the group’s stdyes / no / by the batch’s stdyes: Z.ai reasoning RL; no: Aleph Alpha, DeepSeek V3.2 (as written), Xiaomi (as written), Z.ai agentic RL; batch std (one scalar per step, so it rescales the step, not the prompts): MistralAleph Alpha cites Dr. GRPO; Mistral gives none
Baselinegroup mean / leave-one-outmean: DeepSeek, Xiaomi, Z.ai, Mistral, Aleph Alpha, Moonshot (K2.5); leave-one-out weighted by rewarded tokens: PoolsidePoolside ablated against GRPO and GSPO, no numbers published
Groups with no signaldrop / keep somedrop: Xiaomi, Mistral, Aleph Alpha (which keeps all-pass groups whose format penalties differ)Mistral: “the batch size counts only trajectories that contribute a gradient”
Loss aggregationper token over the batch / per sequence / per prompttoken: Aleph Alpha, MiniMax-M1’s CISPO; sequence: DeepSeek V3.2, GLM-5 reasoning RL; per group in proportion to rollouts: Mistral; prompt: Xiaomi (text; its Eq. 1 prints a token mean)Xiaomi: prompt-mean “prevents response length from growing too quickly”

None reports an ablation of normalisation. Subtracting the group mean is the leave-one-out baseline (RLOO, Kool et al., 2019) scaled by (G−1)/G(G-1)/G, so the two differ only in Poolside’s weighting. Dropping a zero-signal group leaves the gradient’s direction unchanged; it changes the normaliser and, when dropped groups are re-sampled, the mix of prompt difficulties.

Stale rollouts

A synchronous step waits for its slowest rollout, and agentic rollouts are long-tailed: across Xiaomi’s 25 data sources, mean rollout duration varies 66-fold. NVIDIA writes that rollout time “is in turn dominated by a small fraction of straggler generations”. While they finish, the trainer and most inference GPUs sit idle.

Splitting one batch into several gradient steps, as PPO and DeepSeek V3.2 do, already makes every step after the first off-policy. The larger source is to stop waiting. In asynchronous RL, rollout workers generate continuously, the trainer takes finished rollouts as they arrive, and new weights are pushed to the engines while generation continues. Of the 16 labs, ten say how their rollouts are scheduled, and eight call their RL asynchronous: Aleph Alpha, DeepSeek (from V4.1), Mistral, NVIDIA, Qwen (in a model-card line), Thinking Machines, Xiaomi and Z.ai (for agentic RL only). Poolside runs the same way, with weight transfers that “run asynchronously”, without using the phrase for its RL. Moonshot calls its framework synchronous and gets stale data through partial rollouts: in Kimi K3 the generation phase “pauses as soon as a fraction λ∈(0,1) of trajectories completes”, and the rest resume under new weights. Moonshot runs rollout and training on the same GPUs in turn, so generation must stop for training (in DeepSeek V4.1, also co-located, “training preempts ongoing rollouts”); separate pools avoid the pause and pay in weight transfers.

In Aleph Alpha’s first setup, each weight update “waited a median of 221 s for running generations, and took another 217 s to load”. RDMA writes from every trainer rank (direct copies between GPU memories), updating in flight instead of draining, and a broadcast within each node “cut the weight update time down to about 17 s on comparable runs, which enabled weight updates after every training step”, 26 times less (computed). Mistral pushes weights “at each training step”, Poolside “every 2 optimizer steps”, Z.ai every KK updates (unstated), and Z.ai also resets the optimizer after each push, without a measurement.

The reports give staleness in different forms:

LabStalenessWhat it is
NVIDIA1 (2 for one Nemotron 3.5 Super teacher)cap, in policy updates (“maximum trajectory age”)
Xiaomi4, plus a per-source limitstated setting; units not given
Mistral6average, in steps
Poolside10cap, in optimizer steps; weights pushed every 2, so at most 5 engine versions (computed); never reached
Aleph Alphamedian 10.0; cap 42measured, from a group’s oldest start; groups over the cap evicted
Z.ai (agentic)τdrop a rollout if its oldest version is more than τ behind; value not printed
DeepSeek V4.1bounded“we can bound the maximum off-policy ratio”; value not given

Units differ, so does where the age is counted from, and so does how far a step moves the weights: 2,048 sequences at Aleph Alpha, about 7,000 usable trajectories at Mistral, 25,088 at Xiaomi (1,568 prompts times 16), under different optimizers and learning rates. Aleph Alpha explains why staleness rises when a run grows: it is roughly a rollout’s duration divided by the trainer’s step time. In its worked example a 60-minute code rollout with 2-minute steps is 30 steps stale, and doubling the GPUs at a fixed batch size makes it 60, past the cap of 42. “In some of our runs, code groups exceeded the staleness limit in this way after we scaled up the run.” Its remedy is to grow the batch with the GPU count.

staleness (weight updates during the rollout):01–23–56–1011–2021+slot idletrainer updatefirst batchover the cap, discardedstill running

Synchronous: 18 rollouts finished per hour; rollout slots idle 76% of the time; the trainer works 5% of it.

Asynchronous: 76 rollouts finished per hour; rollout slots idle 0%; the trainer waits 7.7 min for its first batch, then steps every 2 min.

Staleness: mean 4.2, maximum 30 (the 60-minute rollout; 60 / 2 = 30).

First batch: mean length 3.5 min, against 9.7 min over all rollouts.

A toy with invented lengths: a fixed log-normal sample, scaled so the longest equals the tail. The 12 asynchronous slots stand for a fleet that keeps the trainer busy, so after its first batch it steps every step time.

Figure 2.A toy. Synchronously, each step lasts as long as its slowest rollout and the other slots wait. Asynchronously, nothing waits and more rollouts finish per hour, but a rollout is roughly its duration divided by the step time in versions stale: with a 60-minute rollout and 2-minute steps, 30. The first batches are the rollouts that finish first, which are the short ones.

Asynchrony also biases which rollouts arrive first. DeepSeek V4.1: “shorter sequences tend to complete first and thus dominate the initial training batches”, especially early in training; it discards early short samples, and Xiaomi replays saved samples at start-up and after recovery. Restarting interrupted rollouts from scratch biases the same way (DeepSeek V4’s reason to resume them), and masking rollouts that hit a turn or time limit drops long ones by rule.

When new weights arrive mid-rollout, the engine holds a KV cache: the attention keys and values of every earlier token, kept so that each new token doesn’t recompute them (00). Labs do three different things with it.

  • Reset: Poolside. “Weight broadcasts trigger an inference-side KV-cache reset, so the cache can never mix tokens encoded under different weight versions”, and an update waits for the current rollout step, so “from each rollout’s perspective the policy is piecewise-constant in time”.
  • Rebuild: Xiaomi, which rebuilds the cache after each update and uses large batches to amortise the cost. Rebuilding, a prefill, is one forward pass over the whole context, up to 150K tokens here.
  • Keep: Mistral, Aleph Alpha, DeepSeek V4.1 and, by implication, Kimi K3, whose paused rollouts keep their KV into the next iteration. Mistral: “its later tokens may attend to KV entries computed by earlier policies.” Aleph Alpha adds that its prefix cache also survives updates, so even a rollout started after an update can read an older policy’s keys and values for its prompt.

With a kept cache, the engine’s probability for a later token came from no single policy. The ratio wt=πθ/μw_t = \pi_\theta/\mu still corrects that token, since μ\mu is what the engine actually sampled with; what loses its meaning is πold\pi_{\text{old}}, and with it the two-factor split. Poolside resets so that its ratio’s denominator is one weight version; Mistral and Aleph Alpha, which keep the cache, take theirs to μ\mu, and DeepSeek V4.1 and Kimi K3 don’t say. No lab measures how much the mixing enlarges wtw_t.

Corrections

Once wt≠1w_t \ne 1, the update can use the per-token weight, which has high variance when a token is far likelier under one policy than the other, or limit it, which bounds the variance and adds bias. Labs use four kinds of rule.

Clip the weight, keep the token

CISPO (MiniMax-M1, 2025) came from a failure of PPO’s clip: rare tokens such as “However” and “Recheck”, which “often serve as “forks” in reasoning paths”, were clipped out after the first of 16 updates per batch, since a small change in a small probability is a large ratio. CISPO keeps every token and clips only its weight, treated as a constant:

ℓt=− sg⁡(min⁡(wt,C)) A log⁡πθ(yt),\ell_t = -\,\operatorname{sg}\big(\min(w_t, C)\big)\, A \,\log \pi_\theta(y_t),

with sg⁡\operatorname{sg} the stop-gradient; without the clip it is the per-token importance-weighted gradient. Mistral (CISPO-style REINFORCE loss) sets C=3C = 3 “with no lower bound”, on wtw_t, so one weight covers both factors. Poolside, which pairs it with leave-one-out advantages, clips to [0,5][0, 5]; its printed equation omits the stop-gradient. In expectation, clipping at CC gives the on-policy gradient with each token counted at probability min⁡(πθ,Cμ)\min(\pi_\theta, C\mu) instead of πθ\pi_\theta: tokens the trainer now favours count as less likely, and the mass they lose goes to no other token. Truncated importance sampling (TIS, Yao et al., 2025) caps the engine factor ρt\rho_t instead and keeps PPO’s clip on utu_t; the name is used loosely, and DPPO’s “PG-TIS” is CISPO.

Mask outside a ratio band

IcePop (Ring-1T, 2025) multiplies each token by M(ρt)M(\rho_t), with ρt=πold/μ\rho_t = \pi_{\text{old}}/\mu the engine factor alone: M(ρ)=ρM(\rho) = \rho inside a band [ρlo,ρhi][\rho_{\text{lo}}, \rho_{\text{hi}}] and 0 outside. Staleness is left to PPO’s clip on utu_t. Ring-1T used [0.5,5][0.5, 5]. In the same paper’s preliminary runs on Ring-mini-2.0, a 16.8B model, that band masked 1 to 2 tokens per thousand, and the narrower [0.5,2.0][0.5, 2.0] “immediately destabilizes training”. GLM-5 sets ρhi=2\rho_{\text{hi}} = 2, exactly that band, for its synchronous reasoning RL and reports no such trouble; neither tested the other’s setup. Xiaomi masks on the single ratio wtw_t, with one band for positive and one for negative advantages, “both initialized to [0.2, 5.0] and tuned independently at runtime based on policy entropy”. NVIDIA’s Nemotron 3 uses “masked importance sampling” with bounds outside our corpus, and Moonshot’s K2.5 (also outside it) masks on the log-ratio.

For agentic RL, where tracking πold\pi_{\text{old}} across weight updates would need “an extensive history of model checkpoints”, Z.ai drops it. Its direct double-sided importance sampling (DDIS) weights each token by wtw_t inside (1−εℓ,1+εh)(1-\varepsilon_\ell, 1+\varepsilon_h) and masks it outside, on both sides; the bounds aren’t printed. That ratio is the importance weight itself. Z.ai credits the “controlled degree of off-policy bias” it accepts to reusing the rollout log-probabilities, but what that gives up is PPO’s separate anchor; the bias comes from the mask. A band, unlike PPO’s one-sided clip, also drops tokens whose update would shrink the mismatch, and since the advantage is one number per rollout, deleting tokens is not neutral even in expectation: a positive rollout moves probability toward the kept tokens, a negative one toward the deleted ones.

Mask on a probability difference

Aleph Alpha’s binary total-variation (B-TV) mask comes from DPPO (Qi et al., 2026), which argues that a ratio is the wrong measure of how far a token moved. A token going from 10−410^{-4} to 10−210^{-2} has ratio 100 and moves almost no probability mass; one going from 0.99 to 0.80 has ratio 0.81, inside any band, yet “removes 0.19 probability mass from the dominant token”. So the mask tests πθ(yt)−μ(yt)\pi_\theta(y_t) - \mu(y_t): with a positive advantage it masks a token whose trainer probability has risen more than 0.2 above the engine’s, with a negative one a token that has fallen more than 0.2 below it. Moves back toward μ\mu are never blocked.

That leaves rare tokens: one going from 0.01 to 0.10 moves little mass, so B-TV keeps it at weight 10. The gradient on a token’s own logit is wt(1−πθ(yt))w_t(1 - \pi_\theta(y_t)), close to wtw_t for a rare token, and Aleph Alpha saw per-token gradients that “reached into the thousands”. It bounds that product at cmax⁡=2c_{\max} = 2, capping the effective weight at cmax⁡/(1−πθ(yt))c_{\max}/(1 - \pi_\theta(y_t)). A plain cap such as CISPO’s, it argues, limits the weight “regardless of the training probability, and thus ignore[s] the softmax factor” 1−πθ1 - \pi_\theta. The mask and the bound together act on “roughly 0.01 % of tokens per step”.

Mask whole negative rollouts

DeepSeek V3.2’s off-policy sequence masking drops a rollout whose advantage is negative and whose mean log⁡(μ/πθ)\log(\mu/\pi_\theta) over its tokens, an estimate of the drift, exceeds a threshold δ\delta. The mask sits inside GRPO’s clipped objective, and DeepSeek’s “old” probability “denotes the sampling probability directly returned by the inference framework”, so PPO’s clip already bounds positive rollouts by wtw_t. A negative token that the trainer now finds ten times likelier still passes at full weight. DeepSeek’s evidence is that the mask “improves stability in certain training scenarios that would otherwise exhibit instability”, with no numbers. Zeroing a negative rollout raises its advantage to 0, an optimistic bias toward the rollouts it masks.

Shown: “rare token gained” (engine probability μ = 0.01, trainer probability πθ = 0.10) with a negative advantage.

weight w = trainer probability πθ / engine probability μ = 0.10 / 0.010 = 10.00

for the Kolibri bound: 1 − πθ = 0.90, so the effective weight is capped at cmax / (1 − πθ) = 2 / 0.90 = 2.22

Three rules pass a weight of 10.0: per-token weight, B-TV, DeepSeek's clip and mask.

A toy for one token at θ = θold, so w = πθ / μ is pure engine mismatch. IcePop’s PPO clip on u is then inactive; DeepSeek V3.2’s clip is taken against the engine probability, so it applies on top of its mask. DeepSeek prints neither δ nor its clip ε: 0.5 and 0.2 are illustrative.

Figure 3.A toy for one token. With no mismatch every rule is the ordinary policy gradient. Away from it they part: clipping caps the weight, a ratio band drops the token, a probability-difference mask drops it only if a lot of mass moved, and DeepSeek's mask drops only negative rollouts the trainer now finds less likely. Try "rare token gained" with a negative advantage: three rules still pass a weight of 10, which is what Kolibri's gradient bound is for.

None of these is new to RL. Truncated weights are V-trace’s in IMPALA (Espeholt et al., 2018), built for the same lag between actors and learner; keeping a behaviour policy apart from a proximal anchor, as IcePop’s two ratios do, is the decoupled PPO objective of Hilton et al. (2022); DPPO’s total variation is the measure in TRPO’s original bound. What is new is the engine as a second source of μ\mu.

The comparisons that exist are each on one model, and they disagree. IcePop’s paper reports beating TIS on a 16.8B model; the R3 paper (below) finds TIS helps plain GRPO but scores 1.6 to 1.7 points lower added to routing replay (two settings, one run each, best checkpoint); DPPO has two truncated variants “collapse prematurely”, one of them CISPO (its “PG-TIS”), in a 1.5B test it shares, with its authors, with the FP16 paper below.

KL to a reference

Classical RLHF penalises the KL divergence from the starting model to limit drift into reward hacking, estimated from sampled tokens. Schulman’s note on approximating KL(πθ∥πref)\mathrm{KL}(\pi_\theta \| \pi_{\text{ref}}) from samples of πθ\pi_\theta compares three per-sample estimators, with q=πref(yt)/πθ(yt)q = \pi_{\text{ref}}(y_t)/\pi_\theta(y_t): k1=−log⁡qk_1 = -\log q (unbiased, high variance), k2=12(log⁡q)2k_2 = \tfrac12 (\log q)^2 (biased, lower variance) and k3=(q−1)−log⁡qk_3 = (q - 1) - \log q (unbiased, always positive). GRPO puts k3k_3 in the loss with β=0.04\beta = 0.04.

DeepSeek V3.2 noticed that its tokens come from the engine, not from πθ\pi_\theta, and that plain k3k_3 fails “particularly when the sampled tokens have substantially lower probabilities under the current policy than the reference policy”. Its corrected estimate multiplies k3k_3 by πθ/πold\pi_\theta/\pi_{\text{old}}, which in V3.2 is wtw_t, so that “the gradient of this KL estimator becomes unbiased”. DeepSeek adds that “for certain domains, such as mathematics, applying a relatively weak KL penalty or even omitting it entirely can yield improved performance.”

Our derivation, checked numerically: used as a loss with samples from πθ\pi_\theta, k3k_3 has gradient (1−q)∇log⁡πθ(1-q)\nabla\log\pi_\theta, whose expectation is the gradient of KL(πref∥πθ)\mathrm{KL}(\pi_{\text{ref}} \| \pi_\theta), the divergence with its arguments swapped, while k2k_2 and DeepSeek’s weighted k3k_3 give the gradient of KL(πθ∥πref)\mathrm{KL}(\pi_\theta \| \pi_{\text{ref}}). Schulman’s note is about values, not gradients. With samples from a stale or mismatched μ\mu, only the weighted form stays exact.

Mistral uses k2k_2 with β=0.005\beta = 0.005, only “for a subset of data sources prone to over-optimization”. Z.ai removes the reference KL from reasoning RL “to accelerate RL improvement”, and Aleph Alpha uses none, as DAPO had argued. Aleph Alpha and Moonshot penalise drift from the rollout policy instead: Aleph Alpha adds λ(log⁡wt)2\lambda (\log w_t)^2 with λ=10−3\lambda = 10^{-3} on every token, and Kimi K2.5 (outside our corpus; K3 follows its algorithm) has −λ(log⁡πθ/πold)2-\lambda(\log \pi_\theta/\pi_{\text{old}})^2. Both are k2k_2-shaped penalties on the importance weight: a trust region in penalty form, anchored to the rollout policy. K3 still streams a reference model’s weights during RL without saying what for. NVIDIA, Xiaomi and Poolside don’t state a reference KL either way.

Precision

Trainers compute in BF16, a 16-bit format; engines can serve in FP8, which halves the bytes each generated token reads. FP8 weights and an FP8 KV cache raised Aleph Alpha’s rollout throughput “by a factor of two to three in our runs”. The cost is a larger engine mismatch: μ\mu now comes from rounded weights, πold\pi_{\text{old}} from unrounded ones. Two labs disagree on what that does.

Aleph Alpha ran FP8 rollouts against an unrounded BF16 trainer and “observed RL training diverge in this configuration” (no curve given). Its fix is quantisation-aware training (QAT) on the trainer: the same FP8 rounding as the engine, computed in BF16. A residual remains: “the median training–inference mismatch |log rt| is roughly three times that of BF16 serving” (their rtr_t is our wtw_t). Poolside tested the same arrangement before release and “observed no measurable hit to stability”, yet kept BF16 inference weights for the release runs of Laguna M.1 and XS.2, “as it was not obvious how this mismatch would play out over the many-day horizon of an RL run”; it later did RL “in FP8 precision” for Laguna S 2.1. The runs differ in length, hardware, correction rule and cache policy, so the disagreement is unresolved.

Of the five other labs that state rollout precision, four quantise both sides alike: Mistral (FP8 inference “to improve generation speed and match the trainer”), Xiaomi (experts in MXFP4, a 4-bit format, with the same rounding applied to the trainer’s weights after every update), Moonshot (“During RL, rollout and training share the same quantization scheme”) and DeepSeek V4 (FP4 QAT, and “native FP4 quantized weights” at rollout). Z.ai uses FP8 for rollouts without saying what the trainer does. An InfiXAI study (LMSYS blog, 2025) found that unified FP8 “effectively eliminates train–inference inconsistency caused by quantization error”.

Matched formats still leave some mismatch, because “none of the kernels are batch-invariant” (Aleph Alpha): sums taken in a different order round differently. NVIDIA (Nemotron 3.5 Super) and Aleph Alpha compute the output head in FP32; MiniMax-M1, which introduced this, reports the correlation of training and inference token probabilities rising from about 0.9 to about 0.99. DeepSeek V4 uses “batch-invariant and deterministic kernel libraries to ensure bitwise reproducibility across training and inference”. An outside paper (Qi et al., 2025) argues BF16 itself is the cause and that FP16, with three more mantissa bits, “effectively eliminates this mismatch”; no lab here reports trying it.

Expert routing

In a mixture-of-experts model a router picks a few experts per token per layer (C2). The pick is a top-k, and as Xiaomi puts it, “numerical differences between the engines can flip discrete expert selections”. The trainer then computes the token’s probability through experts the engine never used. On Qwen3-30B-A3B, the R3 paper measured “approximately 10% of the routers select different experts” between SGLang and Megatron, and “94% of tokens select a different expert in at least one” layer; the MoE model’s mismatch was 2.4 times the dense Qwen3-8B’s (computed from its values).

Figure 4.A flipped top-k makes the trainer score a token through experts the engine never used. Routing replay records the engine's expert indices and reuses them in the training pass; the gate weights are still computed from the trainer's router logits, so the router keeps learning. On one model in the R3 paper, 94% of tokens had at least one layer where the two engines disagreed.

Routing replay records which experts the engine chose for every token and reuses them in the trainer’s forward pass. Five of the eight labs in the table below use it (all eight train MoE models; Moonshot, Poolside and Z.ai don’t state it): DeepSeek, which calls it Keep Routing and has used it “since DeepSeek-V3-0324”; Xiaomi, whose researchers proposed it as Rollout Routing Replay (R3); Mistral (“router replay”); Aleph Alpha (“Routed-Expert Replay”); and NVIDIA, in Nemotron 3.5 Super. The gate weights are recomputed from the trainer’s router logits, so the router keeps learning, at a rollout latency cost “below 3%” in the R3 paper. On one model, replay halved the measured mismatch and kept alive runs in which plain GRPO crashed between steps 60 and 120. DeepSeek V4.1 concatenates the routing of each segment for rollouts that span checkpoints. GSPO (Zheng et al., 2025) addresses a related flip inside the trainer, between πold\pi_{\text{old}} and πθ\pi_\theta: it weights each rollout by the geometric mean of its token ratios and reports that this “has obviated the need” for replay there, on one model, without testing the engine gap.

Replay makes the trainer use the engine’s experts, but the router’s weights still change. On MiMo-V2.6-Pro, at decoder layer 9 (384 experts), Xiaomi measured that over the first 20 RL steps the coefficient of variation of expert load rose from 0.78 to 2.0 and the share of cold experts (load below a tenth of the mean) from 0.5% to 22%, about 84 of 384 (computed), while with the router frozen load stayed near its initial level. Restoring the router’s pre-RL values at step 20 brought load back “while benchmark performance remains unchanged”. Freezing the router is evidence about load and trainability, not scores (load balancing is C3); NVIDIA freezes it in one Nemotron 3.5 teacher. GLM-5’s sparse-attention indexer, another top-k (T3), had the same problem: a fast non-deterministic top-k kernel “caused drastic performance degradation during RL after only a few steps”, so Z.ai switched to the deterministic torch.topk and froze the indexer during RL.

Sampling and tokens

Rollouts are often sampled with top-p: the engine keeps only the most likely tokens whose probabilities add up to pp and renormalises over them. The trainer, computing πθ\pi_\theta over the full vocabulary, gives those tokens a smaller probability than μ\mu did. The ratio is the nucleus’s mass, at least 0.97 at top-p 0.97, so no ratio rule ever fires on it, yet ∣log⁡wt∣|\log w_t| of up to 0.03 (computed) is about the size of the whole mismatch Aleph Alpha measures over its run. DeepSeek’s Keep Sampling Mask records the truncation set during sampling and applies it to πθ\pi_\theta in training, “ensuring both policies share identical action subspaces”; Xiaomi does the same. Aleph Alpha doesn’t truncate in RL (“We do not use top-p or top-k truncation”), at the price of training at top-p 1 while evaluating and serving at top-p 0.97 and top-k 128.

A multi-turn agent’s text goes back into the next prompt, and re-tokenising it can change the tokens: Mistral’s example is “the” followed by “re” becoming “there”. The trainer then scores tokens the engine never produced. Token-in-token-out keeps the sampled token IDs and passes only new text through the tokenizer: Z.ai finds it “critical for asynchronous RL training because it preserves exact action-level correspondence”, and Poolside and Xiaomi use it too, as does Moonshot’s K2.5. Mistral keeps re-tokenisation and handles the mismatch with FA4’s blockwise attention. Aleph Alpha declines token-in-token-out on purpose: “This choice is motivated by downstream usage during deployment, where the text might be re-encoded between turns.” Where the tokens no longer match, it splits the rollout and trains each piece as a separate segment, which “significantly increases the number of segments and thus cumulative tokens for some groups”.

Shorter rollouts

NVIDIA also shortens rollouts for some agentic tasks. PivotRL (Yi et al., 2026) runs GRPO on single turns from expert-trajectory states where sampled actions disagree in outcome; 71% of randomly chosen turns gave no signal. On software-engineering (SWE) tasks it reached 32.67%, a level end-to-end RL also reached, with about 4 times fewer rollout turns (computed from 133K and 542K); both were below the SFT baseline’s 37.40 on SWE-Bench Verified, while PivotRL beat SFT on three other agent benchmarks. Its states come from the expert, so the policy never trains on situations its own earlier turns create. Nemotron 3 Ultra uses it alone for the terminal and conversational tool-use teachers, and before a final end-to-end SWE-RL stage for the SWE teacher.

Scale

Lab, runHardwarePer stepTotal
Mistral Large 43,104 GPUs at peak, trainer to generator 1 : 2.5about 7,000 usable trajectories5.36M rollouts, 162.5B output tokens; up to 26,000 concurrent rollouts
MiMo-V2.6-Pro (Xiaomi)“thousands of GPUs”25,088 sequences, 2.7B to 3.7B training tokens30 steps shown in 123 hours (Fig. 12); 2.6 million US dollars for RL post-training
Kolibri (Aleph Alpha)256 B300: 96 trainer, 144 inference, 16 judge2,048 sequences1,000 steps, 16,900 B300-equivalent GPU-hours; 25,000 sandboxes on average
Nemotron 3 Ultra (NVIDIA)GB200 nodes“global batch size of 8192, with each sample generating 16 rollouts” (prompts or trajectories not stated)not stated

Token counts differ in kind: Mistral’s 162.5B are generated output tokens, Xiaomi’s are training tokens including context (“roughly 110K–150K tokens per sequence”). Kimi K3 keeps each 1M-context RL experiment “within a few hundred GPUs”, and Thinking Machines scaled Inkling’s RL “to over 30M rollouts” in two long runs. Mistral, Aleph Alpha, Poolside, NVIDIA and Z.ai (agentic) keep separate rollout and training pools; Moonshot and DeepSeek V4.1 co-locate; Xiaomi doesn’t say.

What this can’t tell you

DeepSeek V4.1 writes “we refrain from introducing novel post-training algorithms” and credits “the scale, diversity, and verifiability of synthesized data and environments” with “essentially all of the observed gains”. The same report adds sample-level dispatch, concatenated routing replay, a bound on the off-policy ratio and a mask for stale tokens, so a release without “novel post-training algorithms” still carries much of this article’s machinery.

No lab publishes an ablation ranking one correction against another on its shipped model. Tried-and-dropped choices mostly go unrecorded; DeepSeek V4.1’s two rejected dispatch schemes (“severe oscillations in training metrics”, “stalled easily on long-tail samples”) are a rare exception. And the evidence is lopsided: Aleph Alpha’s report is the most detailed public account of most of these choices.

Comparison

MechanismWhat it repairsWhat it costs
Asynchronous RL, partial rolloutsGPUs idle behind long-tailed rolloutsstaleness; short rollouts first; weight transfers or pauses
Staleness capvery old samplesdiscarded rollouts, mostly long ones, so a bias against long solutions
Clip the weight, keep the tokenvariance from high-weight tokenstokens the trainer now favours count as less likely
Mask outside a ratio bandtokens with large mismatchdrops tokens on both sides of the band; band to tune, and labs disagree on its width
Mask on probability differencelarge moves in probability massrare tokens keep large weights, so it needs a gradient bound
Mask negative rolloutsoff-policy failuresoptimistic bias; one threshold
Reference KLdrift from the startslows learning; estimator choice matters
Same quantisation both sidesFP8/FP4 engine mismatchQAT in the trainer
Routing replayflipped expert choicesrecording and moving expert indices
Router freezingrouter drift and expert overloadrouter stops adapting
Sampling-mask replay, or no truncationtop-p renormalisationstoring sets, or training off the serving distribution
Token-in-token-outre-tokenisation drifttrains on token sequences deployment may not produce

Where the labs sit

From the reports and cards; nothing here is checked against code. A blank means searched and not stated.

LabAlgorithm (lab’s word)BaselineCorrectionStalenessRoutingRollout / trainer precisionKV across updateReference KL
Aleph Alphaunnamed (group-relative)mean, no std; token meanB-TV mask + gradient boundmedian 10, cap 42replayFP8 / FP8 QAT, FP32 headkeepnone ((log⁡w)2(\log w)^2 instead)
DeepSeekGRPOmean (V3.2 formula)PPO clip on ww + negative-rollout mask (V3.2); stale-token mask (V4.1)bounded, value not given (V4.1)Keep Routing since V3-0324FP4 / FP4 QAT (V4)keep (V4.1)corrected k3k_3, weak or none for math (V3.2)
MistralCISPO-style REINFORCEmean, batch stdweight clip at 3average 6replayFP8 / FP8keepk2k_2, 0.005, some sources
MoonshotK2.5’s algorithmmean (K2.5)log-ratio mask (K2.5)partial rollout, no numberMXFP4 experts, MXFP8 activations, both sideskeep (implied)none in K2.5’s objective; K3 streams a reference model, use not stated
NVIDIAasynchronous GRPOmasked importance sampling1 (2 for one teacher)replay (3.5 Super)FP32 head
PoolsideCISPO-style, leave-one-outweighted leave-one-outweight clip [0, 5]cap 10 steps, never hitBF16 + FP8 KV / BF16 (M.1, XS.2); FP8 (S 2.1)reset
XiaomiGRPOmean, prompt-mean lossratio-band mask, entropy-tuned4, per-source limitR3, frozen router (V2.6)MXFP4 / same MXFP4 roundingrebuild
Z.aiGRPO + IcePop; DDIS for agentsmean and std (reasoning), mean (agentic)IcePop band [0.5, 2]; DDIS bandon-policy (reasoning); τ (agentic)deterministic indexer top-kFP8 rollout / not statedremoved (reasoning)

Qwen and Thinking Machines say their RL is asynchronous, and Qwen names GSPO for one stage of Qwen3.5-Omni; neither says more about these choices. Google, OpenAI, StepFun and Tencent publish nothing on them, and MiniMax’s corpus documents have nothing on M3’s RL (CISPO comes from its earlier M1 paper). Liquid AI’s LFM2 used offline preference optimisation; LFM3 adds online RL but describes none of these choices.

Easy to get wrong

  • Treating [0.2,5][0.2, 5] as one rule. Xiaomi applies it to the trainer-over-engine weight; NVIDIA’s Nemotron 3.5 Super prints the same band for a ratio between two snapshots of the model in its distillation stage, a staleness ratio. And the IcePop paper found the [0.5,2][0.5, 2] that GLM-5 ships unstable, on its 16.8B Ring-mini-2.0.
  • Comparing mismatch numbers. Mistral’s average k2k_2 of at most 5×10−35 \times 10^{-3}, Aleph Alpha’s median forward KL of 2.4×10−32.4 \times 10^{-3} per token with staleness included, and NVIDIA’s unquantified “more than an order of magnitude” reduction are different statistics on different models. At small mismatch the estimators roughly agree, but a mean is not a median.
  • Taking router freezing as a quality gain. Xiaomi’s evidence is about load.

Check yourself

A group of 16 rollouts has 1 pass and 15 fails (rewards 1 and 0). What advantage does each get under the original GRPO and under DeepSeek V3.2’s written formula? How much more does this prompt count under the original?

The mean is 1/16=0.06251/16 = 0.0625. V3.2 subtracts it: the pass gets +0.94+0.94, each fail −0.06-0.06. The original divides by the standard deviation; with the population one (DeepSeekMath doesn’t say which), 0.0625×0.9375=0.242\sqrt{0.0625 \times 0.9375} = 0.242: +3.87+3.87 and −0.26-0.26. Every advantage is scaled by 1/0.242=4.11/0.242 = 4.1, against 2.0 for a group that half passes, so this hard prompt gets about twice the relative weight of an even one. With the sample standard deviation the pass would get +3.75+3.75.

Poolside caps staleness at 10 optimizer steps and pushes weights to the engine every 2 steps. At most how many engine versions old can a rollout be, and why can’t you compare that with NVIDIA’s cap of 1?

At most 5 versions (10 / 2), since the engine only runs every second set of weights. NVIDIA counts policy updates, and the labs’ steps differ in batch size, learning rate and optimizer, so neither number says how far the weights moved.

A token has engine probability 0.01 and trainer probability 0.10, and its rollout has a negative advantage. What gradient coefficient does it get from Mistral’s clipped weight, GLM-5’s IcePop band, and Aleph Alpha’s B-TV mask with and without its gradient bound?

The weight is 10. Mistral’s is min⁡(10,3)=3\min(10, 3) = 3, so −3-3. GLM-5’s band [0.5,2][0.5, 2] excludes 10, so the token is masked: 0. The B-TV mask keeps it, since with a negative advantage it masks only tokens whose trainer probability fell more than 0.2 below the engine’s: −10-10. The gradient bound then caps the effective weight at cmax⁡/(1−πθ)=2/0.9=2.22c_{\max}/(1 - \pi_\theta) = 2/0.9 = 2.22, so about −2.2-2.2. The interactive figure reproduces these.

In the map

Methods

Taxonomy nodes

Further reading

  • Shao et al., DeepSeekMath, arXiv:2402.03300, and Schulman et al., PPO, arXiv:1707.06347: GRPO and the clipped surrogate.
  • Espeholt et al., IMPALA, arXiv:1802.01561, and Hilton et al., Batch size-invariance for policy optimization (decoupled PPO), arXiv:2110.00641: V-trace’s truncated weights for lagging actors, and separating the behaviour policy from the proximal anchor.
  • MiniMax-M1, arXiv:2506.13585: CISPO and the FP32 head.
  • Liu et al., Dr. GRPO, arXiv:2503.20783, and Yu et al., DAPO, arXiv:2503.14476: the length and difficulty biases, clip-higher and dynamic sampling, with ablation tables.
  • Qi et al., Rethinking the Trust Region (DPPO), arXiv:2602.04879: the case against ratio clipping and the B-TV and B-KL masks.
  • Ring-1T, arXiv:2510.18855: IcePop, its compounding-mismatch argument and its band sensitivity test.
  • Ma et al., Rollout Routing Replay, arXiv:2510.11370, and Zheng et al., GSPO, arXiv:2507.18071: replay’s design and measurements, and sequence-level ratios with replay between policy versions.
  • Qi et al., FP16, arXiv:2510.26788, and the InfiXAI FP8 post (lmsys.org/blog/2025-11-25-fp8-rl/): where numeric mismatch comes from.
  • Schulman, Approximating KL Divergence (joschu.net/blog/kl-approx.html); Kimi k1.5, arXiv:2501.12599 (partial rollouts); Yi et al., PivotRL, arXiv:2603.21383.
  • The labs’ own accounts: Aleph Alpha’s Kolibri report §3.2 and Appendix I (the most detailed), Mistral Large 4 report §4.3 to §4.4, MiMo-V2.6 report §4 to §6, Laguna report §4.4, GLM-5 §3.2 and §4.1, DeepSeek V3.2 §3.1 and V4.1 §5.
↑