Open Models Almanac
Methodspost-trainingreinforcement learning algorithm

single source · post-training

Direct Double-sided Importance Sampling

Direct Double-sided Importance Sampling is a token-level importance sampling method for asynchronous RL that reuses rollout log-probabilities as a direct behavior proxy, avoiding separate old-policy inference.1 It replaces PPO's asymmetric clipping with a double-sided trust region, masking tokens outside the interval from gradient computation to prevent instability from extreme policy divergence.2

Trade-off It accepts a controlled degree of off-policy bias in exchange for avoiding historical policy tracking while boosting training stability.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. reuses the log-probabilities generated during rollout as a direct behavior proxy — GLM-5: from Vibe Coding to Agentic Engineering
  2. Instead of the asymmetric clipping used in standard PPO, we restrict the trust region to — GLM-5: from Vibe Coding to Agentic Engineering
  3. reusing rollout log-probabilities accepts a controlled degree of off-policy bias to circumvent the need for historical policy tracking while boosting training stability — GLM-5: from Vibe Coding to Agentic Engineering

Filed under post-training › reinforcement learning algorithm