specific method · filed under post-training
Off-Policy Sequence Masking
Off-Policy Sequence Masking is a GRPO modification that masks negative-advantage sequences whose policy has diverged too far from the sampling policy.1 It adds a binary mask to the GRPO loss, keyed on a KL-divergence threshold, to stabilize training against off-policy updates from mini-batching and training-inference inconsistency.2
Trade-off It improves stability in training scenarios that would otherwise be unstable.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
we mask negative sequences that introduce significant policy divergence, as measured by the KL divergence between the data-sampling policy and the current policy
— DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelswe introduce a binary mask into the GRPO loss
— DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelsimproves stability in certain training scenarios that would otherwise exhibit instability
— DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
Also called Off-policy sequence masking for GRPO.