Open Models Almanac
Methodspost-trainingreinforcement learning algorithm

specific method · filed under post-training

Group-relative advantage estimation

Group-relative advantage estimation computes the advantage of a segment by subtracting the group mean reward from its training reward.1 This method calculates the mean reward across a group of sequences to provide a relative advantage for each segment.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. Given a groupG of sequences yi, we compute the group-relative advantage of each segment as Ai,e = Ri,e− ¯R — tech report
  2. The mean reward ¯R is computed on a sequence level, over the Ej segments of each sequence yj, to prevent segment splits from overweighting individual sequences — tech report

Also called group-relative advantages, Group-relative reward advantages.

sources
2
models
2
labs adopt it
2
strongest
core

Documented in

Aleph Alpha
Mistral AI

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 1core 1

Kolibri-1

Aleph Alphaadopted1 source

we compute the group-relative advantage of each segment as

coreoptimizationin KolibriAleph Alpha

Mistral-Large-4

Mistral AIadopted1 source

A rollout’s advantage is its reward minus the group mean, and advantages are standardized across the batch.

usedunclearin Mistral Large 4Mistral AI
↑