Open Models Almanac
Methodspost-trainingreinforcement learning algorithm

specific method · filed under post-training

Off-Policy Sequence Masking

Off-Policy Sequence Masking is a GRPO modification that masks negative-advantage sequences whose policy has diverged too far from the sampling policy.1 It adds a binary mask to the GRPO loss, keyed on a KL-divergence threshold, to stabilize training against off-policy updates from mini-batching and training-inference inconsistency.2

Trade-off It improves stability in training scenarios that would otherwise be unstable.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we mask negative sequences that introduce significant policy divergence, as measured by the KL divergence between the data-sampling policy and the current policy — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
  2. we introduce a binary mask into the GRPO loss — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
  3. improves stability in certain training scenarios that would otherwise exhibit instability — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models

Also called Off-policy sequence masking for GRPO.

sources
2
model
1
lab adopt it
1
strongest
used

Documented in

Evidence

3 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 3

DeepSeek-V3.2

DeepSeekadopted2 sources

we mask negative sequences that introduce significant policy divergence, as measured by the KL divergence between the data-sampling policy πold and the current policy πθ.

usedtraining objectivein DeepSeek-V3.2DeepSeek
2 more from this model

we mask negative sequences that introduce significant policy divergence, as measured by the KL divergence between the data-sampling policy and the current policy.

usedoptimizationin DeepSeek-V3.2DeepSeek

Off-Policy Sequence Masking To improve the efficiency of RL systems, we typically generate a large batch of rollout data, which is subsequently split into multiple mini-batches for several gradient update steps.

usedoptimizationin DeepSeek-V3.2DeepSeek
↑