Open Models Almanac
Methodspost-trainingreinforcement learning algorithm

implementation detail · filed under post-training

Group Relative Policy Optimization

Group Relative Policy Optimization is a reinforcement learning algorithm that optimizes the policy model by maximizing a clipped objective on a group of responses sampled from the old policy for each question.1 It estimates each output's advantage by normalizing the outcome reward within the group, subtracting the group's average reward from that output's reward, and adds a KL penalty against a reference policy.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. GRPO optimizes the policy model by maximizing the following objective on a group of responses sampled from the old policy given each question — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
  2. The advantage of is calculated by subtracting the average reward of the group from the reward of output — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models

Also called GRPO, GRPO (Group Relative Policy Optimization), GRPO algorithm, GRPO reinforcement learning training, Group Relative Policy Optimization (GRPO), RL with GRPO.

sources
16
models
9
labs adopt it
5
strongest
core

Documented in

Evidence

18 quotes from 16 sources, one card per model, each opening with its strongest.

How sources treat it
optional 1used 16core 1

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAadopted4 sources

Our training procedure largely follows the asynchronous GRPO algorithm with the stability optimizations proposed in NVIDIA (2026)

usedoptimizationin Nemotron 3 UltraNVIDIA
3 more from this model

The model underwent multi-environment reinforcement learning using asynchronous GRPO (Group Relative Policy Optimization)

usedoptimizationin Nemotron 3 UltraNVIDIA

asynchronous GRPO algorithm with the stability optimizations proposed in nvidia2026nemotron3superopen

usedoptimizationin Nemotron 3 UltraNVIDIA

asynchronous GRPO (Group Relative Policy Optimization)

usedtraining objectivein Nemotron 3 UltraNVIDIA

DeepSeek-V4-Pro

DeepSeekadopted3 sources

For the RL stage, we implemented the Group Relative Policy Optimization (GRPO) algorithm

usedpost trainingin DeepSeek-V4 specialist modelsDeepSeek
3 more from this model

Reinforcement Learning (RL) is applied using Group Relative Policy Optimization (GRPO)

usedpost trainingin DeepSeek-V4 seriesDeepSeek

independent cultivation of domain-specific experts (through SFT and RL with GRPO)

usedpost trainingin DeepSeek-V4 seriesDeepSeek

Reinforcement Learning using Group Relative Policy Optimization (GRPO)

usedtraining objectivein DeepSeek-V4DeepSeek

DeepSeek-V4-Flash

DeepSeekadopted2 sources

For the RL stage, we implemented the Group Relative Policy Optimization (GRPO) algorithm

usedpost trainingin DeepSeek-V4 specialist modelsDeepSeek
2 more from this model

Reinforcement Learning (RL) is applied using Group Relative Policy Optimization (GRPO)

usedpost trainingin DeepSeek-V4 seriesDeepSeek

independent cultivation of domain-specific experts (through SFT and RL with GRPO)

usedpost trainingin DeepSeek-V4-FlashDeepSeek

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑