specific method · filed under optimization
Sinkhorn-balanced update
A momentum-based update followed by Sinkhorn balancing, used for embedding tables and the prediction head.
Also called Momentum-based Sinkhorn-balanced optimizer.
- source
- 1
- model
- 1
- lab adopt it
- 1
- strongest
- default
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 1default 1
Documented in
Evidence
2 spans quoted from the sources, strongest treatment first.
we instead optimize the Engram embedding tables, token embedding, and prediction head using a momentum-based update followed by Sinkhorn balancing.
defaultunclearin DeepSeek-V4.1-FlashDeepSeek
use Sinkhorn-balanced update for all embeddings and prediction head.
usedunclearin DeepSeek-V4.1-FlashDeepSeek
Filed alongside
Other methods under optimization :: optimizer.
MuonPer-Head MuonAdamWCategory-specific assignment of Muon and AdamWSplit fused gradients before orthogonalizationNesterov momentumAdam without weight decay for the N-gram embedding tableAdamW for attention and GDN output gatesAdamW for gated-residual low-rank projectionsAdamW for input embeddings and output headAdamW for the MoE routerAsynchronous Micro-Group pipelineCanzonaCUDA graph capture of the optimizer stepEight-step Newton–Schulz iterationHybrid Muon and AdamW parameter-group optimizer assignmentHybrid Newton–Schulz iterationsHybrid optimization with Muon and AdamMoonlight-style learning-rate scalingMuon orthogonalization accuracy refinementMuon restricted to two-dimensional linear-map weightsMuon SplitMuownPeer-to-peer shard retrieval for Muon orthogonalization