Open Models Almanac
Methodspost-trainingreinforcement learning algorithm

methods · post-training

reinforcement learning algorithm

75 methods filed at this node or below it, from the sources of 23 models.

In this branch 75

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 75

Group Relative Policy Optimization core · 16 sources · 18 quotes
Mixed-domain reinforcement learning core · 5 sources · 5 quotes
Groupwise Advantage Redistribution core · 3 sources · 4 quotes
Chain-of-thought reinforcement learning core · 2 sources · 2 quotes
Group-relative advantage estimation core · 2 sources · 2 quotes
Dropping zero-advantage rollout groups default · 2 sources · 2 quotes
Reinforcement learning used · 20 sources · 23 quotes
Abstention training used · 2 sources · 2 quotes
IcePop used · 2 sources · 3 quotes
Keep Sampling Mask used · 2 sources · 2 quotes
Off-Policy Sequence Masking used · 2 sources · 3 quotes
Rollout Routing Replay used · 2 sources · 3 quotes
59 single mentions
CISPO-style REINFORCE objective core · 1 source · 1 quote
Difficulty-bucket and pass-rate curriculum core · 1 source · 1 quote
Keep Routing core · 1 source · 1 quote
Merlin and Morgana masking algorithms core · 1 source · 1 quote
Penalty Module core · 1 source · 1 quote
Advantage shaping used · 1 source · 1 quote
Checkpoint merging for successive RL runs used · 1 source · 1 quote
Concatenated routing replay used · 1 source · 1 quote
Direct Double-sided Importance Sampling used · 1 source · 2 quotes
Domain-specialized RL experts used · 1 source · 1 quote
Domain-specific GRPO training used · 1 source · 1 quote
Dynamic abstention-reward calibration used · 1 source · 1 quote
End-to-end GRPO on agentic tasks used · 1 source · 1 quote
Group-based rollout sampling used · 1 source · 1 quote
Group-relative length penalty used · 1 source · 1 quote
Group-wise policy optimization used · 1 source · 1 quote
GRPO with masked importance sampling used · 1 source · 1 quote
GSPO used · 1 source · 1 quote
Hierarchical penalty escalation used · 1 source · 1 quote
Incremental reinforcement learning used · 1 source · 1 quote
Interaction-Aligned RL used · 1 source · 1 quote
Length-based RL reward adjustment used · 1 source · 1 quote
Mask re-rendered context during training used · 1 source · 1 quote
Off-policy drift reduction used · 1 source · 1 quote
Omitting advantage normalization used · 1 source · 1 quote
PivotRL used · 1 source · 2 quotes
PPO-style policy-ratio clipping used · 1 source · 1 quote
Progressively complex task distributions used · 1 source · 1 quote
Quality-weighted advantage redistribution used · 1 source · 1 quote
Reinforcement-learning pipeline used · 1 source · 1 quote
RL with rubric and claims graders used · 1 source · 1 quote
RLCEF used · 1 source · 1 quote
SAO with compaction used · 1 source · 1 quote
SFT and GRPO for domain specialization used · 1 source · 1 quote
SFT checkpoint initialization for RL used · 1 source · 1 quote
Single mixed-domain RL run used · 1 source · 1 quote
Single-harness reinforcement learning used · 1 source · 1 quote
Tool-error step penalty used · 1 source · 1 quote
Moonlight scaling optional · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelmethods
DeepSeek-V4.1-Flash usedCheckpoint merging for successive RL runs usedConcatenated routing replay usedEffort-conditioned subgroup reward centering usedReinforcement learning mentioned—
GLM-5.3 usedSAO with compaction used—
Hy3 usedGroup Relative Policy Optimization usedReinforcement learning used—
MiMo-V2.6-Flash coreAsynchronous Group Relative Policy Optimization coreGroupwise Advantage Redistribution corePenalty Module coreDropping zero-advantage rollout groups defaultAdvantage shaping usedDomain-specific GRPO training usedDynamic sampling of informative rollout groups usedFlagged-token advantage penalty and masking usedFreezing the MoE router during reinforcement learning usedGroup-relative length penalty usedHierarchical penalty escalation usedQuality-weighted advantage redistribution usedReplay top-k/top-p sampling candidate sets during training usedRollout Routing Replay usedSingle-harness reinforcement learning usedZero-reward correction for externally supported trajectories used—
NVIDIA-Nemotron-3-Ultra-550B-A55B usedAbstention training usedDynamic abstention-reward calibration usedGroup Relative Policy Optimization usedIcePop usedNegative-advantage penalties for malformed reasoning and tool calls usedPivotRL usedPPO-style policy-ratio clipping usedReinforcement learning usedReinforcement learning from verifiable rewards used—
DeepSeek-V4-Flash usedGroup Relative Policy Optimization used—
Alpamayo 1.5 / 2 Super coreRL-based post-training framework with complementary rewards coreGroup Relative Policy Optimization usedReinforcement learning used—
DeepSeek-V3.2 coreGroup Relative Policy Optimization coreKeep Routing coreMixed-domain reinforcement learning coreImportance-sampling-corrected unbiased KL estimate usedKeep Sampling Mask usedOff-Policy Sequence Masking usedReinforcement learning used—
DeepSeek-V4-Pro usedGroup Relative Policy Optimization usedReinforcement learning used—
GLM-5.2 usedDirect Double-sided Importance Sampling usedGroup Relative Policy Optimization usedGroup-wise policy optimization usedIcePop used—
Inkling usedReinforcement learning usedRL for calibration with proper scoring rules usedRL with rubric and claims graders used—
Kimi K3 coreMulti-domain, multi-effort reinforcement learning coreDomain-specialized RL experts used—
Kolibri-1 coreDifficulty-bucket and pass-rate curriculum coreGroup-relative advantage estimation coreMerlin and Morgana masking algorithms coreDynamic sampling of informative rollout groups usedLarge-scale reinforcement learning for long-horizon tasks usedMask re-rendered context during training usedOmitting advantage normalization used—
LFM3-1.7B usedMixed-domain reinforcement learning usedSFT and GRPO for domain specialization used—
Laguna-S-2.1 coreCISPO with length-weighted leave-one-out group-relative advantages coreAgentic RL task mix with a shared tool-use API usedRLCEF usedTool-error step penalty usedMoonlight scaling optional—
MiMo-V2.5 usedGroup Relative Policy Optimization usedReinforcement learning usedRollout Routing Replay used—
MiMo-V2.5-Pro usedReinforcement learning used—
MiMo-V2.6-Pro corePenalty Module coreDropping zero-advantage rollout groups defaultAdvantage shaping usedAsynchronous Group Relative Policy Optimization usedDomain-specific GRPO training usedDynamic sampling of informative rollout groups usedFlagged-token advantage penalty and masking usedFreezing the MoE router during reinforcement learning usedGroup-relative length penalty usedGroupwise Advantage Redistribution usedHierarchical penalty escalation usedQuality-weighted advantage redistribution usedReplay top-k/top-p sampling candidate sets during training usedRollout Routing Replay usedSingle mixed-domain RL run usedSingle-harness reinforcement learning usedZero-reward correction for externally supported trajectories used—
Mistral-Large-4 coreCISPO-style REINFORCE objective coreImportance weighting for asynchronous policy lag coreKeep importance-weighted tokens in the objective corePer-token weight normalization by rollout group coreStop-gradient upper-clipped importance weighting coreDropping zero-advantage rollout groups usedGroup-based rollout sampling usedGroup-centered batch-wide advantage normalization usedGroup-relative advantage estimation usedMiMo-style segment-level advantage correction usedOff-policy drift reduction usedReinforcement learning usedReinforcement-learning pipeline usedSFT checkpoint initialization for RL used—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B usedAbstention training usedAsynchronous Group Relative Policy Optimization usedClipped-range importance weighting with stale-token masking usedEnd-to-end GRPO on agentic tasks usedGroup Relative Policy Optimization usedGRPO with masked importance sampling usedLength-based RL reward adjustment usedReinforcement learning from verifiable rewards used—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B usedAbstention training usedAsynchronous Group Relative Policy Optimization usedClipped-range importance weighting with stale-token masking usedEnd-to-end GRPO on agentic tasks usedLength-based RL reward adjustment usedReinforcement learning usedReinforcement learning from verifiable rewards used—
Qwen3.5-397B-A17B usedGSPO usedInteraction-Aligned RL usedProgressively complex task distributions used—
gpt-oss-120b coreChain-of-thought reinforcement learning coreHelpful-only adversarial reinforcement learning usedIncremental reinforcement learning used—
↑