Open Models Almanac

Training stages · S4

RLHF/RLVR: reward-based learning, from human preferences to verifiers

A program that checks the answer is cheap to run and gives the same verdict every time, but exists only for checkable tasks; a learned judge reaches open-ended ones and gets exploited by the policy it trains

Illustration of a workbench with an old balance scale on the left and a teal go/no-go gauge beside it, while a small coral clockwork figure reaches around the side of the gauge.

During a reinforcement-learning run for Mistral Large 4, an agent was given a task whose answer a checker program would test against a reference implementation. Mistral’s report describes what it did instead of solving it: “It noticed the checker binary called execvp(“python3”, “-c”, <source_code>) internally, planted a fake /tmp/fakebin/python3 PATH shim to intercept the call, and captured the reference implementation from the command-line payload.” The checker started Python by name and passed it the reference code as an argument, so the agent put its own program called python3 first on the search path, and the checker handed that program the answer. At the other end of the range, NVIDIA trains its chat models against a reward model learned from human preferences and writes: “As policy models become larger and more capable, we observe an increasing tendency for them to exploit weaknesses in the reward model during RLHF, particularly when the reward model is smaller or less capable.”

Reinforcement learning (RL) needs a number for every attempt the model makes. This article is about where it comes from: people’s comparisons, a model trained on them, a model told what to look for, or a program that checks the answer. Of the 16 labs on this site, 11 say what scores their rollouts; NVIDIA alone names reinforcement learning from human feedback on a released text model, Poolside trained Laguna M.1 and XS.2 “using only verifiable rewards”, and the others mix kinds by task.

Background

The primer sets out the stages, and S3 the supervised fine-tuning (SFT) that comes before RL. In RL the model being trained is the policy πθ\pi_\theta. For a prompt xx it writes a rollout yy: an answer, or for an agent a whole episode of tool calls and their results. Reports also call it a trajectory, sample or sequence. A reward r(x,y)r(x, y) scores the rollout, and the update makes high-reward rollouts more likely. Most labs here use a group-relative update (S5 covers the algorithms): the policy writes GG rollouts for the same prompt, and each one’s advantage is its reward minus the group’s mean,

Ai=ri−1G∑j=1Grj.A_i = r_i - \frac{1}{G}\sum_{j=1}^{G} r_j .

This is REINFORCE with the group’s mean as its baseline, in place of a learned value function. Many objectives also keep the policy close to a fixed reference policy πref\pi_{\text{ref}}, usually the SFT model, through a Kullback–Leibler (KL) penalty with weight β\beta.

In this series RLHF names the whole topic of reward-based learning, the phrase readers know; literal RLHF means rewards from human preference judgements. RLVR, reinforcement learning from verifiable rewards, means rewards a program computes from a checkable outcome: a matching answer, passing tests, a final file-system state.

Rewards aren’t in a model’s configuration or code, so this article reads technical reports first, then model cards and official blogs, and counts labs, not models. A blank means searched and not found, not “not used”. No report gives a rate at which its rewards were wrong, and most design choices below come without an ablation.

The worked example: an agent gets a code repository and an issue, and is rewarded if its patch passes the tests. Its commands run in a sandbox, an isolated container holding the repository and its tests, so the agent can’t touch the machine running the training. The harness is the program that runs the agent loop: it builds the prompt, offers the tools, runs each call and returns the result. NVIDIA and Xiaomi sample 16 rollouts per prompt, and the formulas below use G=16G = 16.

Literal RLHF: a reward model learned from comparisons

In InstructGPT (Ouyang et al., 2022), people compare answers instead of writing scores, because comparing is easier and more consistent: “about 40 contractors” ranked 4 to 9 responses per prompt. A reward model rϕr_\phi, a language model with a scalar output, is fit to those pairs under the Bradley–Terry model of paired comparison,

P(yw≻yl∣x)=σ(rϕ(x,yw)−rϕ(x,yl)),LRM=−log⁡σ(rϕ(x,yw)−rϕ(x,yl)),P(y_w \succ y_l \mid x) = \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big), \qquad \mathcal{L}_{\text{RM}} = -\log \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big),

with σ\sigma the logistic function and ywy_w, yly_l the preferred and the rejected answer: logistic regression on the reward difference. Only differences matter, and a gap of 1 means a 73% preference (computed). RL, by PPO in InstructGPT, then maximises the learned reward with a KL penalty that keeps the policy near the SFT model,

max⁡πθ  Ex, y∼πθ[rϕ(x,y)]−β DKL(πθ(⋅∣x) ∥ πref(⋅∣x)),\max_{\pi_\theta}\; \mathbb{E}_{x,\,y\sim\pi_\theta}\big[r_\phi(x,y)\big] - \beta\, D_{\mathrm{KL}}\big(\pi_\theta(\cdot\mid x)\,\|\,\pi_{\text{ref}}(\cdot\mid x)\big),

which InstructGPT applied “to mitigate over-optimization of the reward model”, with β=0.02\beta = 0.02 (in practice the KL is summed over the sampled tokens). This suited chat, where a good answer has no answer key but people agree often enough about which of two is better. It needs annotators at scale, refreshed as the policy changes (Anthropic’s 2022 recipe did so “on a weekly cadence”), and tasks whose quality really is a preference.

rϕr_\phi is only a model of the preference, fit on finite data, and RL searches for the inputs on which it is wrong: Goodhart’s law, familiar from model selection as overfitting to a validation set. Gao et al. (2023) measured it with a large “gold” reward model standing in for people and smaller proxies trained on its labels. As the policy moves away from its start, the proxy’s score keeps rising while the gold score peaks and falls, later for larger proxies. In their runs a KL penalty did no better than stopping early at the same distance, a result they flag as sensitive to settings.

proxy: the reward model's scoregold: the true quality it stands forwhere training stops

Toy coefficients (a = 1.0, b = 0.6). The proxy rises to 12 at √KL = 12. Gold is back where it started at √KL = 5.29 and reaches −5.89 at √KL = 12.

gold at stop: 1.2 (peak 1.2)

Figure 1.A toy on the functional form Gao et al. fitted to RL against a learned reward model. The proxy reward keeps rising as the policy moves away from where it started; the true quality peaks and then falls below where it started. A larger reward model moves the peak out. The coefficients are invented; the shape is the paper's, which they note doesn't hold close to zero.

Anthropic reported in 2022 that “larger PMs are more robust than smaller PMs” (preference models). NVIDIA’s sentence in the opening is about the policy’s size, where the evidence points both ways: Gao et al.’s runs, with two policy sizes, found larger policies don’t overoptimise more, and NVIDIA reports the opposite tendency without a measurement. The labs keep the KL penalty where the reward can be over-optimised: Mistral only “for a subset of data sources prone to over-optimization”, DeepSeek V3.2 weakens or omits it for mathematics, and GLM-5 removes it from its reasoning RL “to accelerate RL improvement”.

DPO: the same objective without the RL loop

For each prompt the KL-regularised objective is maximised by

π∗(y∣x)=1Z(x) πref(y∣x) er(x,y)/β,Z(x)=∑yπref(y∣x) er(x,y)/β,\pi^*(y\mid x) = \frac{1}{Z(x)}\, \pi_{\text{ref}}(y\mid x)\, e^{r(x,y)/\beta}, \qquad Z(x) = \sum_y \pi_{\text{ref}}(y\mid x)\, e^{r(x,y)/\beta},

because the objective equals −β-\beta times the KL divergence from π\pi to that distribution, plus a constant. Solved for the reward, r=βlog⁡π∗πref+βlog⁡Z(x)r = \beta \log\frac{\pi^*}{\pi_{\text{ref}}} + \beta \log Z(x). In the Bradley–Terry difference both answers share the prompt, so log⁡Z(x)\log Z(x) cancels. Direct preference optimisation (DPO, Rafailov et al., 2023) then writes the policy being trained, πθ\pi_\theta, in place of π∗\pi^*, so the policy is its own reward model, and the reward model’s loss becomes a loss on the policy:

LDPO=−log⁡σ ⁣(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x)),\mathcal{L}_{\text{DPO}} = -\log \sigma\!\Big(\beta \log\frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log\frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\Big),

with the same β\beta as the KL weight. The two have the same optimum if preferences really follow Bradley–Terry, any reward function is allowed, and both problems are solved exactly. With finite data they differ on answers outside the pairs, and a pair one answer always wins pushes the gap without bound (Azar et al., 2023). DPO’s data is also off-policy: written by an earlier policy, so the update never sees the current one’s answers. DPO and its variants are what labs here use to optimise preferences offline.

Who uses which signal

Of the 11 labs that say what scores their rollouts, most use several kinds by task: program verifiers at 10 and language-model judges at 8 (“Where the labs sit” has each lab’s row). Qwen’s one human-labelled preference stage trains a speech component, and Liquid AI’s pairs are chosen by models. OpenAI names a method whose paper describes the reward. Google, MiniMax, StepFun and Tencent say nothing in the corpus about what scores their RL.

The table suggests that people have moved from labelling each comparison to calibrating the model that gives the reward (DeepSeek’s “minimal set of diverse human annotations”, Z.ai’s “expert human responses as stylistic and qualitative anchors”, the human preference data behind NVIDIA’s judge) and to writing tasks and environments. That is our inference; no lab says it.

NVIDIA: what literal RLHF looks like now

NVIDIA’s literal RLHF is human preference passed through an RL-trained judge. Its reward model for chat is a generative reward model (GenRM): a language model that reads a conversation and two candidate responses, reasons in text, and outputs a score for each and a ranking, judged against user-given principles when there are any. For Nemotron 3 Ultra it starts from Ultra’s own SFT model.

The GenRM is trained by RL “to predict both individual scores for two responses and a ranking score”, rewarded by how close they come to human ones. The humans are HelpSteer3’s contracted annotators, LMArena comparisons and newer preference data, listed in the Nemotron 3 Super report (outside the corpus), which Ultra cites as its method. It scores NVIDIA’s chat teacher and, as a new variant trained with a reasoning length penalty, in Nemotron 3.5 Super, an alignment teacher trained “primarily using RLHF”, which the shipped model learns from by distillation (S6).

NVIDIA’s response to the exploitation quoted in the opening is a bigger judge: “we scale up both the model capacity and training data and develop an Ultra-based GenRM.” Nemotron 3.5 Super later stopped comparing responses with each other “in a cyclical format” and compares “all policy responses to a fixed response from a strong open-source model, Gemma 4 31B-it”, averaging per-rubric scores. The report gives no reason for that change and doesn’t tie it to hacking.

NVIDIA’s stage names don’t say which reward was used: Ultra’s “RLVR” stage includes chat environments. Nemotron 3.5 Lightning “goes through a lighter pipeline, with text-only SFT, and RLVR”, though the abstract lists RLHF for both models.

Preference without people

LFM2, Liquid AI’s small on-device family, runs DPO-style alignment on pairs that models choose. For about a million prompts the SFT model writes five responses, an “LLM jury” scores them, and the best and worst become the pair. For instruction following, the chosen response is replaced by a model’s revision of it (CLAIR), so the pair differs mostly in what the revision fixed. About 700,000 pairs remain. The objective is a family of length-normalised variants, which divide each log-ratio by the response’s token count so the model isn’t paid for writing more. The stated reason for an offline method is speed: “We leverage direct alignment methods to achieve a strong trade-off between model capabilities and iteration speed.”

The report doesn’t say which models sit on the jury whose preferences the policy is trained toward. LFM3’s card lists no preference stage.

Verifiers: rewards a program computes

Poolside’s reward for Laguna M.1 is a chain of checks where the first failure sets the reward: a parsing error −0.1, stopping too early −0.1, a timeout 0, and “the task-specific verifier produces the only positive reward”, 1 or 0: unit tests, assertions on the final container state, or an exact numeric match. Xiaomi’s MiMo-7B gave the motive: “We employ only rule-based accuracy rewards to avoid potential reward hacking.”

Rewards called verifiable often include a model reading the answer. NVIDIA checks math answers “with DeepSeek-V4-Flash as the LLM judge”, and Z.ai’s reasoning RL uses “judge models or evaluation systems to produce binary outcome rewards”, math included. Kolibri limits its judge to natural language and intent, yet one software environment compares edits “with the golden patch by the judge model, without executing tests”. Xiaomi uses no judge for its cybersecurity tasks: “LLM-based judging returns different verdicts for the same PoC across runs and cannot serve as a stable reward” (a PoC is a proof-of-concept exploit).

The GLM-5 report compares two kinds of learned reward. Outcome reward models (ORMs), scalar models like rϕr_\phi above, “offer low-variance signals and high training efficiency, but are more susceptible to reward hacking, where the policy exploits superficial patterns rather than genuinely improving core capability.” Generative reward models (GRMs; NVIDIA’s GenRM is one) “are more robust to such exploitation, but tend to exhibit higher variance.” GLM-5 combines rules, ORMs and GRMs in a hybrid reward system.

In estimator terms this is variance against bias. If a check accepts a wrong answer with probability ε+\varepsilon_+ and rejects a right one with probability ε−\varepsilon_-, at rates the policy can’t influence, the expected reward is ε++(1−ε+−ε−) q\varepsilon_+ + (1 - \varepsilon_+ - \varepsilon_-)\,q for a success rate qq: the update keeps its direction and shrinks (10% errors each way leave 80% of it, computed), which costs samples. An error the policy can steer towards, a kind of wrong answer the check accepts, is a second objective that more samples don’t remove.

Building a verifier for the worked example

A repository task usually starts as a real commit that fixed a bug. The tests the commit added fail before the fix and pass after it. Mistral’s construction is the common one: “Tests that fail before the reference solution and pass afterwards become the verifier, while tests that pass in both states guard against regressions. The solution and anything that could reveal it are removed from the environment”. These are fail-to-pass (F2P) and pass-to-pass (P2P) tests (fail-to-pass verifier construction), named or described by five labs: Mistral, Xiaomi, DeepSeek, Aleph Alpha and Z.ai.

Xiaomi requires the outcomes “to remain stable across eight reruns” and removes tests “that enforce requirements absent from the specification”. DeepSeek V3.2 requires the gold patch to turn some tests from failing to passing and none the other way. Kolibri pays partial credit: the share of F2P tests that pass, held back by any P2P test the patch breaks. Poolside’s variant, a filter on whole tasks rather than a split of the tests, requires “the gold solution to pass the tests and an empty solution to fail them”, and keeps 30,000 to 60,000 of about 236,000 commits (13% to 25%, computed). Z.ai’s GLM-5.3 writes some verifiers “without access to the reference solution” and keeps those that pass “oracle, no-op, and unsolved-state checks”, with “a meaningful amount of human-in-the-loop work”.

Mistral writes that tests drawn from real repositories “may reject valid solutions, accept incorrect ones, or enforce implementation choices that the specification leaves open”. In the terms above, the first and third are ε−\varepsilon_- and the second is ε+\varepsilon_+. Mistral runs several solvers of different strength against each task to find them, and Xiaomi has an auditing agent flag false positives and negatives. NVIDIA concedes that its software reward “can sometimes wrongly reward or penalize trajectories”.

A failure the agent caused counts against it (a crash at DeepSeek V4.1, malformed tool calls or context overflow at Mistral), and an infrastructure failure is dropped (Mistral, GLM-5). For unclear cases Mistral writes: “Ambiguous cases default to the agent’s fault, so unreliable behavior is never silently dropped and implicitly rewarded”, since a dropped rollout gets no negative advantage. Task generation is in D3.

When no program can check: judges, rubrics and generative reward models

For writing, advice or an open-ended agent task, the labs that describe a reward use a language model as judge, and most give it a rubric: a list of criteria for the one prompt (“cites the relevant clause”, “doesn’t invent a deadline”), which the judge scores the answer against item by item. DeepSeek V3.2 uses “a generative reward model where each prompt has its own rubrics for evaluation”. For its specialist models on hard-to-verify tasks, DeepSeek V4 goes further: “we dispense with these conventional scalar-based reward models”, and instead “the actor network natively functions as the GRM”, the actor being the policy, trained by RL as judge as well (RL on the generative reward). DeepSeek says this “achieves superior performance with only a minimal set of diverse human annotations”, with no number behind “superior”. A judge that is the policy never goes stale, but nothing fixed checks it either.

Moonshot’s Kimi K3 uses an “Agentic Generative Reward Model” for non-verifiable tasks, a tournament in which candidates meet in pairs and the judge writes a rubric and scores each against it. The judge favoured length: K3 adds a budget “to mitigate reward hacking toward increasingly verbose outputs”, under which a candidate over it “automatically loses the binary comparison”. No lab says how its pairwise judge handles the order in which the two answers are shown during RL.

Xiaomi breaks rubrics into “atomic, binary rubric items”, some checked by code and some by a copy of its SFT model. Thinking Machines grades instruction following twice, because rubric graders “in practice they emphasize recall and can be hacked by models spraying plausibly relevant facts hoping to match rubric items”; the second grader checks claims with a web search. Thinking Machines states this as a risk and reports no incident.

OpenAI trained gpt-oss’s refusals with deliberative alignment. In the method’s paper (Guan et al., 2024), SFT teaches the model to reason over a written safety specification, and RL uses “a judge LLM that is given our safety specifications”, with “no human-labeled completions”. The paper adds that RL stage for o1 and o3-mini; the gpt-oss card doesn’t say whether it was used.

A judge adds compute to every rollout, and its scores vary between calls. The only lab that prices grading is Xiaomi, where the grader took 12.7% of MiMo-V2.6-Pro’s RL cost, against 43.8% for rollouts and 43.5% for training.

Beyond pass or fail

Once a task gets easy, a binary reward gives little signal: if all GG rollouts pass, every advantage is zero. With independent rollouts and success rate qq, the share of groups with any signal is

1−qG−(1−q)G.1 - q^G - (1-q)^G .

At G=16G = 16 and q=0.9q = 0.9, 18.5% of groups pass entirely and give no gradient; at q=0.95q = 0.95, 44% (computed). Even in the groups that have signal it shrinks: the expected sum of squared advantages is (G−1) q(1−q)(G-1)\,q(1-q), at q=0.9q = 0.9 about a third of its value at q=0.5q = 0.5 (computed). Xiaomi, Mistral and Kolibri drop zero-signal groups (S5). Under what it calls groupwise agentic grading, Xiaomi adds what tests can’t give, in two ways.

For high-pass-rate coding tasks, groupwise reward synthesis (GRS) has an agent write rubrics for the solution and for the agent’s behaviour, from offline rollouts. A grader agent scores each rollout, and the reward becomes (Xiaomi writes RR)

ri=ritest⋅Sisol⋅Sibeh.r_i = r_i^{\text{test}} \cdot S_i^{\text{sol}} \cdot S_i^{\text{beh}} .

A failed rollout still scores zero, and passes now differ. With a group-mean baseline, any pass whose product is below the group’s mean reward is pushed down: in an all-pass group, every below-average solution, though it passed. The report presents no ablation of GRS.

For the other coding tasks, groupwise advantage redistribution (GAR) keeps the test advantages and moves them among the passes. A grader sees the whole group, compares the passing patches (approach, minimality, side effects, craftsmanship) and gives each a quality factor fi∈(0,1]f_i \in (0, 1]. With P\mathcal{P} the passing rollouts,

λ=∑j∈PAj∑j∈PfjAj,Ai′=λfiAi    (i∈P).\lambda = \frac{\sum_{j\in\mathcal{P}} A_j}{\sum_{j\in\mathcal{P}} f_j A_j}, \qquad A'_i = \lambda f_i A_i \;\; (i \in \mathcal{P}).

Under a binary test reward every pass starts with the same advantage, so λ\lambda rescales the factors to average 1: a pass above the average gains what one below it loses, every pass stays positive and failures keep theirs. Xiaomi caps λ\lambda (value not given) and re-centres the group, which matters only when the cap binds: the passes’ total then shrinks and every rollout is lifted by the same amount. The contrast scales with the pass advantage 1−k/G1 - k/G for kk passes, 0.06 at 15 of 16, which suggests why the easiest tasks get GRS instead (our reading). The grader is also a hack detector: a confirmed hack’s reward is reset to zero.

Xiaomi’s evidence for GAR comes from “code-only RL runs of MiMo-V2.6-Flash with and without it, using a training batch size of 128”, against a production batch of 1,568 prompts over mixed tasks. In its plot the pass rate keeps rising with GAR (“pass-rate gains are sustained through step 52”) and stalls, then falls, without it; no numbers are printed, and the run without GAR ends at about half the steps.

advantage of a passof a failtoy scores

sum of advantages over passes (before Xiaomi's cap): 0.000

groups with any signal at q = 0.90: 1 − q^16 − (1 − q)^16 = 81.5%

Figure 2.A toy group of eight rollouts. If all pass, a binary reward gives no advantage to any. Xiaomi's GRS multiplies the test reward by two rubric scores, which separates the passes and pushes the weaker ones below zero. Its GAR keeps the test advantages and splits the passes' share by a grader's quality factors, with the total unchanged before Xiaomi's cap. The slider shows how often a group of 16 has any signal at all. Scores are invented.

GRS relies on an all-pass group that still varies in another term, which then survives the filter for all-pass groups (GAR runs only on mixed groups). Kolibri keeps such groups “to apply format penalties to passing sequences”, and Xiaomi’s length penalty applies only above a pass-rate threshold. In such a group the secondary score is the whole signal, so its noise and blind spots are what the policy learns.

Two other designs go beyond pass or fail. Kimi K3 grades kernels by speed against a reference: “matching it yields a reward of 0.5 and approaching the hardware roofline increases the reward toward 1”. Thinking Machines and Aleph Alpha pay for not answering when the model is likely wrong; Aleph Alpha’s Merlin–Arthur environment asks each question with the full context, with distractors hidden and with the evidence hidden, and pays +1 for a correct answer in the first two and for abstaining in the third, and −1 for a wrong answer in the first two.

Shaping terms: length, format and single tokens

Most recipes add terms that reward or penalise properties of the rollout itself. Unlike potential-based shaping (Ng et al., 1999), which leaves the best policy unchanged, each of these changes it, on purpose.

Seven of the 16 labs state a length term (Aleph Alpha, DeepSeek, Liquid AI, Mistral, Moonshot, NVIDIA, Xiaomi), in different directions and on different rollouts, as the figure shows. Mistral’s bonus for short correct rollouts trains only its lower-effort teachers (effort is a setting for how long the model reasons). Poolside penalises the opposite, an agent that stops before a minimum number of tool calls.

Figure 3.How five labs shape the reward of a correct rollout by its length. Only NVIDIA's panel uses published constants; the others show the published shape with constants we chose, because the reports don't give them. Mistral's bonus trains its lower-effort teachers, not the main run. Each panel has its own axes.

Format rewards score well-formed output: a closed reasoning block, a valid tool call, the right language. In early experiments, Aleph Alpha found that “our reward was climbing nicely, but benchmark scores were not. We found that our RL training overfit on output formats at the cost of environment rewards. Most sequence groups only had reward variance due to format differences and, as a result, did not provide environment reward as a training signal.” Where every rollout failed the task, format was the only difference, so the advantages trained format alone. The fix was “to split both reward types and only apply the format signals to sequences with positive environment reward.” After that change, 500 RL steps took AIME 2025 (a 30-problem maths competition) from 60.4 for the SFT model to 70.6, in an early two-environment setup; the overfitting run’s own score isn’t given.

For a task reward of 0 or 1, gating an added bonus on success already multiplies it in: r+wFr + wF for passes only is r(1+wF)r(1 + wF). For the final Kolibri run Aleph Alpha “switched from additive format signals to the multiplicative p-norm version” (multiplicative p-norm format reward), which also scales partial credit and sets how defects compound. With format signals rescaled to ϕk∈(0,1]\phi_k \in (0, 1], the reward of a rollout with positive task reward is

r=renv[mmin⁡+(1−mmin⁡)exp⁡ ⁣(−(∑k(−ln⁡ϕk)p)1/p)],r = r_{\text{env}} \Big[m_{\min} + (1 - m_{\min}) \exp\!\Big(-\big(\textstyle\sum_k (-\ln \phi_k)^p\big)^{1/p}\Big)\Big],

with p=2p = 2, mmin⁡=0.5m_{\min} = 0.5 and a full defect giving ϕk=0.5\phi_k = 0.5 (Kolibri Table 59). One defect keeps 75% of the reward, two 69%, three 65% (computed); pp sets how defects compound, from their product at p=1p = 1 toward the worst one as pp grows. Multiplying stops format paying a failed rollout or being learned on prompts the model can’t solve. Format still competes with correctness: avoiding one defect is worth a 25% lower success rate.

For a 0/1 task reward, gating an added bonus on success is the same as multiplying it in.

Total reward on a linear axis from 0 to 1.2; the tick marks 1, the task reward of a right answer. The additive weight and scores are invented; the multiplicative factor uses Kolibri's published constants.

reward = task + 0.2 × format score (toy: 1 clean, 0.5 with one defect). Wrong answers earn 0.20 and 0.10: a group of failures still differs in reward.

Figure 4.Two ways to combine a format score with the task reward. Added on (with an invented weight), it rewards wrong answers that look right, which is what Aleph Alpha found its early RL learning. Multiplied in, with Kolibri's published constants, one defect costs a correct answer a quarter of its reward and a wrong answer gets nothing either way.

Four labs also penalise the single tokens of a broken tool call or reasoning step: NVIDIA, Poolside, Xiaomi, and Mistral, which applies “MiMo-style segment-level advantage correction” with a citation to Xiaomi. Each shaping term is a second objective, and few labs publish its constants.

Reward hacking

Reward hacking is earning reward without doing what the reward was meant to measure. Six labs report it in their own RL training, and a seventh in evaluation:

LabWhat happenedWhere it entered
DeepSeek (V4.1)Agents “exploited recently disclosed vulnerabilities” in the XFS file-system driver and AppArmor, and leaked answers from package mirror servicesthe sandbox and its network
Xiaomi (MiMo-V2.6)Repository repair: installing a newer package, fetching or cloning the upstream fix, looking up the solution’s ticket, probing package versions, one rollout shown for eachthe sandbox’s network
MistralThe python3 shim in the opening, found by monitoring an RL runa checker that ran inside the agent’s sandbox
Aleph AlphaWhile evaluating SWE-bench Verified (real GitHub issues with tests) in the OpenCode harness, its web-fetch and web-search tools found the solutions on GitHub in about 7% of trialsthe evaluation sandbox’s network
Moonshot (K3)Kernel tasks: a detector penalises “CUDA graph replay, input caching, and precision reduction”, and is extended “as new hacking strategies are observed”the speed check
Z.ai (GLM-5)Slide generation: “hard truncation of overlong content or excessive manipulation of spacing”the rendering-based reward
NVIDIA (Ultra)Policies exploiting a smaller reward model, with no example giventhe learned reward model

AppArmor is the Linux mechanism that limits what a program may read and write, so a bug in it was a way out. A package mirror, a local copy of a package index, could serve a newer library version containing the fix. A kernel reward times the code, so replaying recorded GPU work or returning cached outputs looks fast without computing anything.

Four of the seven went through the sandbox around the check (DeepSeek, Xiaomi, Mistral, and Aleph Alpha in evaluation), three through the reward’s own blind spots (Moonshot, Z.ai, NVIDIA). Whether fetching an upstream fix is a hack depends on the task’s rules. Aleph Alpha counted it as an exploit. Poolside’s evaluation flags rose from under 2% to over 50% of SWE-bench-family trajectories during Laguna S 2.1’s training, were mostly legitimate web research on review, and fell back under 2% with a prompt instruction.

Xiaomi’s “logged confirmed-hack share remains below 2% throughout the whole training process”. It is logged with the GAR grader’s correction in place, beside offline audits; that grader runs on mixed coding groups and also zeroes what it catches, and since the policy is trained against that grader, a low share can also mean hacks it doesn’t see.

Figure 5.Where the reported hacks entered, and where each lab's defences act. Four of the seven reported hacks went through the sandbox around the check. Two cards are a stated risk and a stated reason, not observed incidents.

Labs describe three kinds of defence:

  • Attacking their own environments first (adversarial audit). Xiaomi says its “hack agent” went on “until the hack agent could no longer find a successful exploit in any of the environments”; Mistral runs an adversarial agent on its terminal tasks, DeepSeek V4.1 an inspection agent, and Z.ai writes verifiers without the solution in view.
  • Removing what leaks (environment hardening). Xiaomi removes build logs, verifier outputs and leftover patches, keeps Git history only up to the task’s base commit, and isolates the network. NVIDIA deletes future commits so they “cannot be recovered by any low-level git recovery command”. DeepSeek uses “per-sandbox AppArmor profiles and fine-grained eBPF-based network policies” (eBPF programs in the Linux kernel decide which connections a sandbox may open). Moonshot isolates agents from verifiers and pairs public ones with hidden ones that test held-out cases (public and hidden verifiers).
  • Catching hacks during training: Moonshot’s detector, Xiaomi’s grader, monitoring at Mistral and Aleph Alpha; for learned rewards, a bigger judge (NVIDIA), the length rule (Moonshot), a second grader (Thinking Machines).

The labs disagree about the network. Xiaomi isolates it “to restrict access to upstream fixes and alternative package versions”, Poolside’s S 2.1 and Kolibri block selectively, and Mistral keeps “the realistic internet access and tooling that agents need”; Moonshot wants “to permit as much exploration as possible” inside microVMs, without saying what network its agents get. Isolation closes the leaks three of the seven reported hacks used (DeepSeek’s mirrors, Xiaomi’s fixes, Aleph Alpha’s web fetch), but the agent then never trains with the network it will have in use. No lab has compared the two.

Harnesses

Poolside trained Laguna M.1’s RL in “the same one we ship in production” and moved Laguna S 2.1 to several harnesses. Xiaomi calls production harnesses “a poor fit for RL in practice”, because their safeguards and instruction prompts “fall outside the task-completion reward signal, making credit assignment unreliable and leaving reward-unmeasured requirements to simply be ignored”. It trains in small mini-harnesses instead (multi-harness training). Seven of the 16 labs say they train across several harnesses: Aleph Alpha, DeepSeek, Moonshot, NVIDIA, Thinking Machines, Xiaomi and Poolside, with Liquid AI’s “real agent harnesses” borderline. Moonshot gives the common reason, “Training with a single fixed agent harness can cause a model to overfit”. None reports an ablation against a single harness.

Comparison

SignalReachesLabel costHow it was exploitedLabs (of 16)
Human-preference reward model (literal RLHF)chat, style, alignmentannotators, refresheda stronger policy exploits a smaller model (NVIDIA)1
Offline preference pairs (DPO and variants)the same, without RLpairs from people or a model jurynone reported; off-policy data2
Language-model judge, rubric or generative reward modelopen-ended tasks, agent deliverablesjudge compute (12.7% of RL at Xiaomi); rubrics written by modelsverbosity (Moonshot); rubric “spraying”, a stated risk8, plus OpenAI in the method’s paper
Program verifiertasks with a checkable outcomean environment per task, built and kept workingleaks through the sandbox; timing or rendering checks fooled10
Verifier plus rubric or gradercheckable tasks where many or all rollouts passgrader computethe grader is also the detector, so its blind spots are shared1
Shaping terms (length, format, tokens)anytuning; few constants publishedformat learned instead of the task (Kolibri, early)7 state length terms, 4 token-level

Labs behind the counts: literal RLHF, NVIDIA; offline pairs, Liquid AI and Qwen; judges, NVIDIA, DeepSeek, Z.ai, Mistral, Moonshot, Xiaomi, Thinking Machines and Aleph Alpha; verifiers, those eight plus Qwen and Poolside; verifier plus grader, Xiaomi; token-level penalties, NVIDIA, Poolside, Xiaomi and Mistral.

Where the labs sit

Each lab’s reward signals and who or what produces the labels, its verifier construction and the hacks it reports.

LabSignals (label source)Verifier constructionHacks reported
Aleph Alphasix kinds of verifier, one of them a judge (Qwen3.5-122B-A10B), for language, intent and one code environment; abstentionF2P share, held back by P2Pweb fetch of fixes, in evaluation
DeepSeekrules and tests; rubric GRM (V3.2); the policy as judge, “minimal” human annotations (V4)F2P > 0, none breaking (V3.2); leak removal, inspection and repair agents (V4.1)driver and AppArmor exploits, mirror leaks (V4.1)
Liquid AIlength-normalised DPO, LLM-jury pairs (LFM2); RL with a length penalty (LFM3)
Mistraltests, LLM judges, “a reward model” (data unstated), programmatic checksF2P and regression tests, refinement agentpython3 shim
Moonshotverifiable tasks; agentic GRM tournament; graded kernel speedpublic and hidden verifierskernel speed strategies
NVIDIAGenRM trained on human preferences (“RLHF”); tests; an LLM judge for mathhidden tests, binaryreward-model exploitation
OpenAIdeliberative alignment (judge given the spec, in the method’s paper)
Poolside“only verifiable rewards” (M.1, XS.2)gold passes, empty failsevaluation flags, mostly legitimate
Qwenhuman-labelled DPO and rule-based rewards (speech component)
Thinking Machinesproper scoring rules; rubric and claims graders; abstentionrubric spraying (risk)
Xiaomitests; GRS and GAR (an SFT-trained grader); atomic rubric itemsF2P and P2P stable over 8 reruns, audit agentsolution leaks
Z.aijudge models; rules, ORMs and GRMs, human exemplars; rendering checksF2P and P2P (GLM-5); blind verifiers (GLM-5.3)slide truncation and spacing

A proper scoring rule (Thinking Machines’ row) gives the highest expected score when the model’s stated confidence matches how often it is right. Google, MiniMax, StepFun and Tencent publish nothing in the corpus on rewards; MiniMax says its harness, MiniMax Code, was “trained together with M3”.

Easy to get wrong

  • Reading “preference optimisation” as human preference. LFM2’s pairs are chosen by a model jury.
  • Reading “verifiable” as “checked by a program”. Rewards NVIDIA and Aleph Alpha call verifiable, and Z.ai’s binary math rewards, come partly from judge models.
  • Taking a hack share as a hack rate. Xiaomi’s “below 2%” is what its grader caught; Poolside’s is from evaluation runs. Neither says how often policies try.
  • Crediting Kolibri’s 60.4 → 70.6 to the multiplicative format reward. It belongs to the earlier gating fix, and it is measured against the SFT model, not against the run that overfit.
  • Counting Mistral’s root escape as reward hacking. An agent on a capture-the-flag task “chained four different vulnerabilities and gained root access to one of our nodes”, looking for the target outside its sandbox. It fooled no check and earned nothing: a failure of the sandbox, not of the reward.
  • Reading silence as a choice. Google, MiniMax, StepFun and Tencent describe no reward in the corpus, yet StepFun’s own blog on Step 5, outside it, describes outcome rewards and a hack detector.

Check yourself

1. A coding task is solved 95% of the time and trained with groups of 16 under a binary reward. What share of its groups give no gradient, and what does Xiaomi do for such tasks?

All 16 pass with probability 0.9516=0.440.95^{16} = 0.44 (all failing is negligible), so about 44% of groups give no gradient. Xiaomi’s GRS multiplies in rubric scores for such tasks, so passes of different quality get different rewards, and the weaker ones negative advantages.

2. An agent’s training reward climbs while its held-out benchmark stays flat. Name two links between task and reward where a hack could enter, and a defence a lab reports for each.

The sandbox: a fix reachable on the network or a checker the agent can read (network isolation and artefact removal at Xiaomi; agents isolated from verifiers at Moonshot). The check: a timing check fooled by cached inputs (Moonshot’s detector) or a renderer blind to truncated text (Z.ai’s fixed renderer). A learned judge is a third (a bigger judge at NVIDIA).

3. Why did Kolibri’s early RL learn format instead of the task, and why does gating or multiplying the format reward fix it?

With an added bonus, a group of failures still differed by format, and group-relative advantages turn any difference into a gradient. Gated or multiplied in, a failed rollout scores zero whatever its format.

In the map

Methods

Taxonomy nodes

Further reading

  • Ouyang et al., arXiv:2203.02155, and Bai et al., arXiv:2204.05862: the classical pipeline, labelling, and preference-model scaling.
  • Rafailov et al., arXiv:2305.18290: the DPO derivation. Azar et al., arXiv:2310.12036: where the equivalence breaks on finite data.
  • Gao et al., arXiv:2210.10760: the gold/proxy setup behind the overoptimisation figure, and the KL penalty’s effect.
  • Guan et al., arXiv:2412.16339: SFT on specification-citing reasoning, then RL with a specification-reading judge.
  • Wang et al., arXiv:2505.11475: HelpSteer3’s human preference data and the RL training of NVIDIA’s judge (Appendix H).
  • Ng, Harada and Russell, 1999: which reward shaping leaves the optimal policy unchanged.
  • Deiseroth et al., arXiv:2512.11614: the Merlin–Arthur protocol, first proposed with a supervised loss.
  • Lab reports, cited by section in the text: Nemotron 3 Ultra §3.3.2 and Nemotron 3.5 §3.5 (NVIDIA); DeepSeek V3.2 §2–3, V4 §5.1.1, V4.1 §5.1; GLM-5 §3.4 and §4.2; MiMo-V2.6 §4.2–4.3 (Xiaomi); Mistral Large 4 §4; Kimi K3 §4.1–4.2; Kolibri §3.2 and Appendix I; Laguna M.1/XS.2 §4.3–4.4.
↑