single source · data curation
Disagreement-based curation of RL post-training data
A data curation method that selects RL post-training samples where the model's implicit reward (from its logits) disagrees with the explicit reward model.1 It computes the model's predicted probability distribution and a Boltzmann distribution implied by the rewards, and treats large divergence between them as marking valuable samples.2
Trade-off Focusing RL updates on this curated set yields higher alignment efficiency and more robust learning dynamics than uniformly sampled data.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
prioritize samples where the model’s implicit reward signal (encoded in its logits) disagrees with the explicit reward model
— Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long TailA large divergence between these two distributions indicates that the model’s internal preference (its implicit reward) conflicts with the externally defined reward signal
— Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailwe achieve both high alignment efficiency and robust learning dynamics compared to uniformly sampled data
— Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail
Also called prioritize these high-disagreement samples.
Filed under data curation › data filtering