DeepSeek-V3.2
DeepSeekBased on p_t,:, we set a KL-divergence loss as the training objective of the indexer
1 more from this model
Based on , we set a KL-divergence loss as the training objective of the indexer
specific method · filed under training objective
KL alignment loss is an auxiliary loss that trains an index branch whose top-block selection is non-differentiable under the main language-modeling loss.1 It works by matching the index branch's score distribution to the main branch's attention distribution on the selected tokens, with the teacher detached from gradients.2,3
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
The top- selection in Equation 7 is non-differentiable, so the language-modeling loss cannot train the index projections directly. We therefore train the Index Branch with a KL alignment loss— MiniMax Sparse Attention
The KL loss gives the Index Branch a direct learning signal by matching its scores to the Main Branch on the selected tokens— MiniMax Sparse Attention
the teacher distribution is detached from gradient computation— MiniMax Sparse Attention
Also called KL-divergence loss, KL divergence loss for indexer, KL loss.
4 quotes from 3 sources, one card per model, each opening with its strongest.
Based on p_t,:, we set a KL-divergence loss as the training objective of the indexer
Based on , we set a KL-divergence loss as the training objective of the indexer
KL loss directly supervises the Index Branch by matching its selection distribution
train the Index Branch with a KL alignment loss
Picked by hand, not extracted: where to read more, not evidence for anything on this page.