specific method · filed under optimization
Per-Head Muon
Per-Head Muon is a variant of the Muon optimizer for attention projections that orthogonalizes each attention head's block separately instead of the full projection matrix.1 It equalizes update scales across heads, since full-matrix orthogonalization lets larger-scale heads dominate the shared update direction.2
Trade-off It yields more balanced learning dynamics and slightly reduces optimizer overhead, since Newton–Schulz on tall per-head blocks is cheaper.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
instead of applying Newton–Schulz orthogonalization to the full Q, K, and V projection matrices, we partition their momentum matrices along the head dimension and orthogonalize each head’s block separately
— k3 tech reportheads with larger gradient or momentum scales dominate the shared update direction, while smaller-scale heads receive insufficiently normalized updates; per-head orthogonalization equalizes the update scale across heads
— k3 tech reportyields more balanced learning dynamics across heads and improves training stability at larger scales. It also slightly reduces optimizer overhead, as Newton–Schulz iterations on tall per-head blocks are cheaper
— k3 tech report
Also called head-wise Muon, Per-Head Muon optimizer.