Open Models Almanac
Methodsdata curationdata filtering

single source · data curation

Disagreement-based curation of RL post-training data

A data curation method that selects RL post-training samples where the model's implicit reward (from its logits) disagrees with the explicit reward model.1 It computes the model's predicted probability distribution and a Boltzmann distribution implied by the rewards, and treats large divergence between them as marking valuable samples.2

Trade-off Focusing RL updates on this curated set yields higher alignment efficiency and more robust learning dynamics than uniformly sampled data.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. prioritize samples where the model’s implicit reward signal (encoded in its logits) disagrees with the explicit reward model — Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail
  2. A large divergence between these two distributions indicates that the model’s internal preference (its implicit reward) conflicts with the externally defined reward signal — Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail
  3. we achieve both high alignment efficiency and robust learning dynamics compared to uniformly sampled data — Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail

Also called prioritize these high-disagreement samples.

Filed under data curation › data filtering