Open Models Almanac
Methodspost-trainingreward modelling

general family · filed under post-training

Generative Reward Model

A Generative Reward Model uses the actor network to function as both the generator and the judge, scoring its own outputs against structured rubrics.1 It addresses hard-to-verify tasks by using structured rubrics to evaluate policy trajectories, fusing evaluation into the generative process.2

Trade-off The method achieves superior performance while requiring only a minimal set of diverse human annotations.3

How this definition was written

Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.

  1. The actor network functions as both the generator and the judge, scoring its own outputs against structured rubrics — DeepSeek V4 Guide: Engram Memory, Training Data Strategy & Release Status (2026)
  2. Instead, to address hard-to-verify tasks, we curate rubric-guided RL data and employ a Generative Reward Model (GRM) to evaluate policy trajectories — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
  3. achieving "superior performance with only a minimal set of diverse human annotations — DeepSeek V4 Guide: Engram Memory, Training Data Strategy & Release Status (2026)

Also called GenRM, Generative Reward Model (GRM), rubric-conditioned generative reward model.

sources
6
models
4
labs adopt it
2
strongest
used

Documented in

Evidence

7 quotes from 6 sources, one card per model, each opening with its strongest.

How sources treat it
used 7

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAadopted3 sources

Generative Reward Models (GenRM) (Wang2025HelpSteer3Preference) with reasoning capabilities help mitigate such reward hacking behaviors

usedoptimizationin Nemotron 3 UltraNVIDIA
2 more from this model

Nemotron 3 Ultra 550B-A55B GenRM: GenRM used for RLHF

usedpost trainingin Nemotron 3 UltraNVIDIA

GenRM used for RLHF

usedpost trainingin Nemotron 3 UltraNVIDIA

DeepSeek-V4-Pro

DeepSeekadopted2 sources

we curate rubric-guided RL data and employ a Generative Reward Model (GRM) to evaluate policy trajectories.

usedpost trainingin DeepSeek-V4DeepSeek
1 more from this model

Generative Reward Model replaces scalar reward models

usedpost trainingin DeepSeek-V4DeepSeek

DeepSeek-V3.2

DeepSeekadopted1 source

We then develop detailed evaluation rubrics across multiple quality dimensions and employ a generative reward model to score responses based on these rubrics.

useddata curationin DeepSeek-V3.2DeepSeek
1 more from this model

For general tasks, we employ a generative reward model where each prompt has its own rubrics for evaluation.

usedtraining objectivein DeepSeek-V3.2DeepSeek

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑