Model techniques map
Techniquesoptimizationoptimizer

specific method · filed under optimization

Per-Head Muon

A Muon variant that partitions attention-head weights or momentum matrices and optimizes or orthogonalizes each head separately.

Also called head-wise Muon, Per-Head Muon optimizer.

sources
4
models
2
labs adopt it
2
strongest
default

How sources treat it

One count per evidence span, weakest treatment to strongest.

used 4default 1

Documented in

DeepSeek-V4.1-FlashKimi K3

Evidence

5 spans quoted from the sources, strongest treatment first.

First, we use head-wise Muon, where Query weights are split by head before applying the Muon update.

defaultunclearin DeepSeek-V4.1-FlashDeepSeek
DeepSeek V41 Tech Report·DeepSeek

we partition their momentum matrices along the head dimension and orthogonalize each head’s block separately.

usedoptimizationin Kimi K3Moonshot AI
k3 tech report·Kimi Team

We optimize the model using the Per-Head Muon optimizer (§ 2.5) together with the weight-clipping mechanism introduced in Kimi K2

usedunclearin Kimi K3Moonshot AI
k3 tech report·Kimi Team

Per-Head Muon optimizer: Learning rate scheduling at the individual attention head level

usedoptimizationin Kimi K3Moonshot AI
Kimi K3 Model Overview: 2.8T Parameters, MXFP4 Quantization, and What the Open Weights Mean for the Community·Hugging Face community blog·2026-07-17

Per-Head Muon extends Muon by optimizing attention heads independently for more

usedoptimizationin Kimi K3Moonshot AI
Kimi K3 Tech Blog: Open Frontier Intelligence·Moonshot AI·2026-07-16

Filed alongside

Other methods under optimization :: optimizer.

MuonAdamWCategory-specific assignment of Muon and AdamWSplit fused gradients before orthogonalizationNesterov momentumSinkhorn-balanced updateAdam without weight decay for the N-gram embedding tableAdamW for attention and GDN output gatesAdamW for gated-residual low-rank projectionsAdamW for input embeddings and output headAdamW for the MoE routerAsynchronous Micro-Group pipelineCanzonaCUDA graph capture of the optimizer stepEight-step Newton–Schulz iterationHybrid Muon and AdamW parameter-group optimizer assignmentHybrid Newton–Schulz iterationsHybrid optimization with Muon and AdamMoonlight-style learning-rate scalingMuon orthogonalization accuracy refinementMuon restricted to two-dimensional linear-map weightsMuon SplitMuownPeer-to-peer shard retrieval for Muon orthogonalization