Open Models Almanac
Methodstraining objectiveauxiliary loss

specific method · filed under training objective

KL alignment loss

KL alignment loss is an auxiliary loss that trains an index branch whose top-block selection is non-differentiable under the main language-modeling loss.1 It works by matching the index branch's score distribution to the main branch's attention distribution on the selected tokens, with the teacher detached from gradients.2,3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. The top- selection in Equation 7 is non-differentiable, so the language-modeling loss cannot train the index projections directly. We therefore train the Index Branch with a KL alignment loss — MiniMax Sparse Attention
  2. The KL loss gives the Index Branch a direct learning signal by matching its scores to the Main Branch on the selected tokens — MiniMax Sparse Attention
  3. the teacher distribution is detached from gradient computation — MiniMax Sparse Attention

Also called KL-divergence loss, KL divergence loss for indexer, KL loss.

sources
3
models
2
labs adopt it
2
strongest
used

Documented in

Evidence

4 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
used 4

MiniMax-M3

MiniMaxadopted1 source

KL loss directly supervises the Index Branch by matching its selection distribution

usedtraining objectivein MiniMax Sparse AttentionMiniMax
1 more from this model

train the Index Branch with a KL alignment loss

usedtraining objectivein MiniMax-M3MiniMax

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑