Open Models Almanac
Methodsoptimizationtraining stability

single mention · optimization

Off-policy sample filtering

Off-policy sample filtering discards training samples that are too far behind the current policy or produced by environment failures.1 A sample is dropped when its oldest rollout model version is stale beyond a threshold, and samples failing due to environment collapse are excluded, with incomplete groups padded or dropped.2,3,4

Trade-off It removes spurious reward noise and improves training stability, at the cost of discarding some trajectories.5,6

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. To filter out these severely off-policy samples, we log the policy weight version used by the rollout engine at generation time — GLM-5: from Vibe Coding to Agentic Engineering
  2. We discard a sample if its oldest rollout version is too stale — GLM-5: from Vibe Coding to Agentic Engineering
  3. exclude samples that fail due to environment collapse — GLM-5: from Vibe Coding to Agentic Engineering
  4. we pad the group by repeating valid samples if the number of valid samples exceeds half of the group size; otherwise, we drop the entire group — GLM-5: from Vibe Coding to Agentic Engineering
  5. This procedure reduces spurious reward noise and improves training stability — GLM-5: from Vibe Coding to Agentic Engineering
  6. This removes trajectories that lag too far behind the current policy — GLM-5: from Vibe Coding to Agentic Engineering

Also called Dropping off-policy and noisy samples.

Filed under optimization › training stability