Open Models Almanac
Methodstraining objectiveauxiliary loss

single source · training objective

Gradient detach

Gradient detach applies stop-gradient to the Index Branch input in a sparse-attention model.1 This blocks the KL alignment loss from updating the Main Branch projections and the backbone, so the auxiliary objective trains only the indexer.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. To isolate the auxiliary objective from the backbone, we apply stop-gradient to the Index Branch input — MiniMax Sparse Attention
  2. so leaves the Main Branch projections untouched; Equation 11 further prevents it from reaching the backbone — MiniMax Sparse Attention

Also called gradient detach for KL loss.

Filed under training objective › auxiliary loss