Open Models Almanac
Methodsoptimizationoptimizer

specific method · filed under optimization

Per-Head Muon

Per-Head Muon is a variant of the Muon optimizer for attention projections that orthogonalizes each attention head's block separately instead of the full projection matrix.1 It equalizes update scales across heads, since full-matrix orthogonalization lets larger-scale heads dominate the shared update direction.2

Trade-off It yields more balanced learning dynamics and slightly reduces optimizer overhead, since Newton–Schulz on tall per-head blocks is cheaper.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. instead of applying Newton–Schulz orthogonalization to the full Q, K, and V projection matrices, we partition their momentum matrices along the head dimension and orthogonalize each head’s block separately — k3 tech report
  2. heads with larger gradient or momentum scales dominate the shared update direction, while smaller-scale heads receive insufficiently normalized updates; per-head orthogonalization equalizes the update scale across heads — k3 tech report
  3. yields more balanced learning dynamics across heads and improves training stability at larger scales. It also slightly reduces optimizer overhead, as Newton–Schulz iterations on tall per-head blocks are cheaper — k3 tech report

Also called head-wise Muon, Per-Head Muon optimizer.

sources
3
model
1
lab adopt it
1
strongest
used

Documented in

Moonshot AI

Evidence

4 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
used 4

Kimi K3

Moonshot AIadopted3 sources

We optimize the model using the Per-Head Muon optimizer (§ 2.5) together with the weight-clipping mechanism introduced in Kimi K2

usedunclearin Kimi K3Moonshot AI
3 more from this model

we partition their momentum matrices along the head dimension and orthogonalize each head’s block separately.

usedoptimizationin Kimi K3Moonshot AI

Per-Head Muon extends Muon by optimizing attention heads independently for more

usedoptimizationin Kimi K3Moonshot AI

Per-Head Muon optimizer: Learning rate scheduling at the individual attention head level

usedoptimizationin Kimi K3Moonshot AI
↑