Open Models Almanac
Methodspost-trainingpreference optimization

single mention · post-training

Offline direct preference alignment

Offline direct preference alignment optimizes a model's responses to preferred over dispreferred ones using direct alignment objectives on a fixed preference dataset.1 It combines length-normalized direct alignment objectives with an offline dataset mixing on-policy samples from the SFT checkpoint and off-policy reference responses.2

Trade-off Direct alignment methods aim to balance model capabilities with fast iteration.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. We leverage direct alignment methods to achieve a strong trade-off between model capabilities and iteration speed — LFM2 Technical Report
  2. we combine a family of length-normalized direct alignment objectives with an offline dataset consisting of both on-policy samples generated from the SFT checkpoint and off-policy reference responses — LFM2 Technical Report
  3. a strong trade-off between model capabilities and iteration speed — LFM2 Technical Report

Also called Preference Alignment.

Filed under post-training › preference optimization