single source · training objective
Gradient detach
Gradient detach applies stop-gradient to the Index Branch input in a sparse-attention model.1 This blocks the KL alignment loss from updating the Main Branch projections and the backbone, so the auxiliary objective trains only the indexer.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
To isolate the auxiliary objective from the backbone, we apply stop-gradient to the Index Branch input
— MiniMax Sparse Attentionso leaves the Main Branch projections untouched; Equation 11 further prevents it from reaching the backbone
— MiniMax Sparse Attention
Also called gradient detach for KL loss.
Filed under training objective › auxiliary loss