Open Models Almanac
Methodspost-trainingreinforcement learning algorithm

specific method · filed under post-training

Importance-sampling-corrected unbiased KL estimate

Importance-sampling-corrected unbiased KL estimate is a correction applied to the K3 estimator to obtain an unbiased KL estimate.1 It uses the importance-sampling ratio between the current and old policy to make the KL estimator's gradient unbiased, which eliminates systematic estimation errors.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we correct the K3 estimator (Schulman, 2020) to obtain an unbiased KL estimate using the importance-sampling ratio between the current policy and the old policy — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
  2. As a direct result of this adjustment, the gradient of this KL estimator becomes unbiased, which eliminates systematic estimation errors, thereby facilitating stable convergence — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models

Also called Unbiased KL Estimate.

sources
2
model
1
lab adopt it
1
strongest
used

Documented in

Evidence

3 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 3

DeepSeek-V3.2

DeepSeekadopted2 sources

we correct the K3 estimator (Schulman, 2020) to obtain an unbiased KL estimate using the importance-sampling ratio between the current policy πθ and the old policy πold.

usedoptimizationin DeepSeek-V3.2DeepSeek
2 more from this model

we correct the K3 estimator (Schulman, 2020) to obtain an unbiased KL estimate using the importance-sampling ratio between the current policy πθ and the old policy πold.

usedtraining objectivein DeepSeek-V3.2DeepSeek

we correct the K3 estimator (Schulman, 2020) to obtain an unbiased KL estimate using the importance-sampling ratio between the current policy and the old policy.

usedoptimizationin DeepSeek-V3.2DeepSeek
↑