single mention · post-training
Offline direct preference alignment
Offline direct preference alignment optimizes a model's responses to preferred over dispreferred ones using direct alignment objectives on a fixed preference dataset.1 It combines length-normalized direct alignment objectives with an offline dataset mixing on-policy samples from the SFT checkpoint and off-policy reference responses.2
Trade-off Direct alignment methods aim to balance model capabilities with fast iteration.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
We leverage direct alignment methods to achieve a strong trade-off between model capabilities and iteration speed
— LFM2 Technical Reportwe combine a family of length-normalized direct alignment objectives with an offline dataset consisting of both on-policy samples generated from the SFT checkpoint and off-policy reference responses
— LFM2 Technical Reporta strong trade-off between model capabilities and iteration speed
— LFM2 Technical Report
Also called Preference Alignment.
Filed under post-training › preference optimization