Open Models Almanac

Optimization · O1

Optimizers: AdamW, Muon, and what each one bounds

Two limits on a training step, one on each weight and one on a layer's output, and what the difference costs in memory, communication and fine-tuning

Illustration of a large ink-blue sheet of cloth on a plain table. Across its left half, dozens of tiny coral pins each hold one point on a short thread. Around its right half runs a single rigid teal hoop. Gold thread lies coiled at the edge.

DeepSeek trained V3 with AdamW and V4-Pro with Muon. Mistral pre-trained Large 4 on AdamW, Xiaomi did the same for MiMo-V2.6, and NVIDIA for Nemotron 3 Nano and Super. Of the 11 labs we found stating a pre-training optimizer, 8 use Muon at their latest model that says. Two of those 8 print a comparison they ran themselves: Moonshot, and on stability Qwen. The rest cite others, report an observation without a number, or give no reason.

The switch has a price. Six of the eight describe something they built so that Muon could run on their training systems. Those that say which weights use it name the linear maps, so a second optimizer or a second rule stays in the run for the rest. Moonshot’s Moonlight paper reports reaching the same loss with about half the compute of AdamW. A Stanford benchmark, sweeping each optimizer up to about 0.5B parameters and extending to 1.2B by a fitted law, found a gain of 1.1× and shrinking.

This article explains what each optimizer does to a gradient and why that makes one of them expensive in a way the other isn’t. It then reads the evidence on pre-training, on the parameters Muon can’t handle, and on what happens when a Muon-trained checkpoint is fine-tuned.

Training recipes aren’t in the configs or modeling code we check, so everything here comes from documents: first the labs’ technical reports, then method papers, each named where it’s used. A blank in a table means we searched and didn’t find it, which says nothing about whether the lab does it. Tokens are pre-training tokens seen, and batch sizes are global, unless a sentence says otherwise.

What an optimizer costs

An optimizer turns a gradient into a weight update. At frontier scale that has a cost in space, in compute and communication, and in convergence.

Plain momentum keeps one running average of the gradient per parameter. Adam keeps two buffers per parameter, and Muon one. On V4-Pro’s 1.6T parameters, at FP32, that’s 12.8 TB against 6.4 TB (computed: 1.6T × 4 bytes × 2 or × 1). The dtype is our assumption: DeepSeek’s V3 report tracks its AdamW moments in BF16, a 16-bit format, which would halve both, and the V4 report doesn’t say.

The 2:1 ratio holds for the state alone. Counting FP32 weights and gradients as well, the per-parameter total falls by about a quarter (computed: 16 bytes to 12). On a large cluster the state is spread thin, and Moonshot’s K2 report calls its per-device footprint “negligible”. No lab names memory as a reason for choosing Muon in general. It motivated one narrower choice: DeepSeek V4.1 adds 196B parameters of n-gram tables and says that applying Adam to them “substantially increases the optimizer-state memory footprint”. Qwen3.8 has a smaller n-gram table and keeps it on Adam.

AdamW touches each element on its own, and NVIDIA calls it “infinitely shardable”: any GPU can update any slice of any matrix. Muon’s update depends on each whole matrix, so that cost needs Muon’s definition first, and comes after it.

Convergence, how far the loss falls per token, is why a lab would accept the second cost. It gets its own section after the mechanisms.

AdamW: one step size per weight

Adam divides each coordinate’s momentum by the root of an average of its squared gradients. With gradient ∇\nabla at one coordinate:

mt=β1mt−1+(1−β1)∇t,vt=β2vt−1+(1−β2)∇t2,w←w−η mtvt+ϵm_t = \beta_1 m_{t-1} + (1-\beta_1)\nabla_t, \qquad v_t = \beta_2 v_{t-1} + (1-\beta_2)\nabla_t^2, \qquad w \leftarrow w - \eta\,\frac{m_t}{\sqrt{v_t}+\epsilon}

This is a diagonal preconditioner: each weight gets its own scale, and weights never influence each other’s scale. Orvieto and Gower give an exact reading, building on Balles and Hennig. When β1=β2\beta_1 = \beta_2 and ϵ\epsilon is ignored, mm is an online estimate of the gradient’s mean, and v−m2v - m^2 is an online estimate of its variance σ2\sigma^2. Since v=m2+σ2v = m^2 + \sigma^2, the step becomes

mm2+σ2=sign⁡(m)1+σ2/m2.\frac{m}{\sqrt{m^2 + \sigma^2}} = \frac{\operatorname{sign}(m)}{\sqrt{1 + \sigma^2/m^2}} .

So at equal betas a coordinate moves by at most η\eta, and by less when its gradient is noisy. (At the common (0.9, 0.95) a run of same-sign gradients can briefly pass that bound.) Orvieto and Gower call this steepest descent inside a trust region set by the signal-to-noise ratio. The name is loose. A classical trust region tests each step and can reject it, while here the step always lands on the boundary, so the content is in the radius, which shrinks as noise grows.

The noise term matters in practice. Keeping only the sign of the momentum, the optimizer known as Signum, needed about a third more budget than Adam for the same loss on their 160M model, even after extensive tuning (computed from their “3/4 of the budget”). They recommend β1=β2\beta_1 = \beta_2 as a default, tested up to 410M parameters. Among the labs, only Mistral states equal betas in pre-training, (0.975, 0.975), and reports “moderate gains”. Others that state betas, such as DeepSeek on its AdamW groups, Aleph Alpha on Kolibri and NVIDIA, use (0.9, 0.95).

Moonlight observes that AdamW’s update RMS (the root mean square of its elements, per unit learning rate) sits around 0.2 to 0.4. Kosson et al. and NVIDIA give a factor of κ\sqrt{\kappa}, with κ=(1−β1)/(1+β1)\kappa = (1-\beta_1)/(1+\beta_1), which comes from averaging: an average of independent noise has variance κσ2\kappa\sigma^2. Combining this with the formula above gives a toy (our construction). For independent gradients with mean μ\mu, variance σ2\sigma^2 and ρ=μ/σ\rho = \mu/\sigma, E[m2]/E[v]=(ρ2+κ)/(ρ2+1)E[m^2]/E[v] = (\rho^2+\kappa)/(\rho^2+1). That splits into a drift ρ/1+ρ2\rho/\sqrt{1+\rho^2}, the part that moves along the gradient on average, and a jitter from noise left in the momentum.

Drift and jitter add as squares: RMS² = drift² + jitter².

A toy, at 1×: RMS 0.249, drift 0.100, floor √κ 0.229.

Figure 1.A toy for one coordinate with independent gradients. When noise dominates, the update RMS sits near the momentum factor, and most of it is jitter: at β = 0.9 and ρ = 0.1 the RMS is 0.25 and the drift 0.10. As a larger batch reduces noise, both rise toward 1, the size of a sign update. The RMS and drift formulas are within about 3% of simulation for Gaussian noise. They fail for intermittent gradients, such as a rare token's.

So the observed 0.2 to 0.4 doesn’t say how much of Adam’s update is noise. At β1=0.9\beta_1 = 0.9 the toy’s floor is 0.23: an RMS of 0.25 would be about 80% noise by squared size, and 0.4 about 30%. Which applies depends on ρ\rho, and we found no measurement of it on a frontier run. The factor κ\sqrt{\kappa} describes one step. The noise in momentum is correlated across steps, so over many steps a coordinate’s random walk comes out about the same size whatever β1\beta_1 is (simulated: about 0.9 per square root of steps at 0.9, 0.95 and 0.975). A larger β1\beta_1 gives a smaller step, and not a smaller random walk.

Weight decay pulls weights toward zero each step. If you add it to the gradient, as L2 regularization, Adam’s division scales it too, and as Loshchilov and Hutter put it, weights with large gradients “do not get regularized as much”. Decoupled weight decay subtracts it outside the preconditioner, so every weight shrinks by the same fraction: w←w−η(m/v+λw)w \leftarrow w - \eta(m/\sqrt{v} + \lambda w). That is AdamW.

AdamW’s costs are the second buffer and a step that depends on noise. Its advantages are that it needs no communication beyond the gradient reduction a training system already does, and that it has no parameter it can’t handle.

Muon: a limit on how far a layer’s output moves

Take a layer y=Wxy = Wx and an update ΔW\Delta W. Its output moves by Δy=ΔW x\Delta y = \Delta W\, x, and that change is what the next layer sees. Bernstein’s derivation of Muon starts here. If an update moves a layer’s outputs too much, the network can destabilize, so pick the update that most reduces the linearized loss (the first-order change, ⟨∇,ΔW⟩\langle \nabla, \Delta W \rangle, the sum of elementwise products) while bounding that output change:

min⁡ΔW ⟨∇,ΔW⟩subject to∥ΔW∥RMS→RMS≤η.\min_{\Delta W}\ \langle \nabla, \Delta W \rangle \quad \text{subject to} \quad \|\Delta W\|_{\text{RMS}\to\text{RMS}} \le \eta .

The norm is the largest factor by which ΔW\Delta W can scale the average entry size of an input, which equals the spectral norm (largest singular value) times din/dout\sqrt{d_{\text{in}}/d_{\text{out}}}. It bounds output change for inputs of RMS at most 1. Write the gradient’s singular value decomposition as UΣV⊤=∑iσiuivi⊤U \Sigma V^\top = \sum_i \sigma_i u_i v_i^\top. Then

⟨∇,ΔW⟩=∑iσi  ui⊤ΔW vi,\langle \nabla, \Delta W \rangle = \sum_i \sigma_i\; u_i^\top \Delta W\, v_i ,

and each ui⊤ΔWviu_i^\top \Delta W v_i is at least minus the spectral norm of ΔW\Delta W. Setting every one to that limit gives

ΔW=−ηdout/din  UV⊤.\Delta W = -\eta \sqrt{d_{\text{out}}/d_{\text{in}}}\; U V^\top .

So the update keeps the gradient’s singular vectors and drops their sizes. UV⊤UV^\top is the orthogonal factor of the polar decomposition (UV⊤)(VΣV⊤)(UV^\top)(V\Sigma V^\top), the nearest matrix to the gradient with all singular values equal to 1. Since the objective is linear, the solution always sits on the boundary: the step has the full allowed size whatever the gradient. This is the normalized steepest descent direction, or in Frank-Wolfe terms the linear minimization oracle of the ball. “Trust region” is the common shorthand, with the same looseness as for Adam. Natural gradient and K-FAC measure output change on average over the data, using the input covariance. This bounds it for the worst input of a given size, and needs no statistics of the activations (our comparison).

Muon applies this to the momentum MM and not to the gradient, so the update is (MM⊤)−1/2M(MM^\top)^{-1/2} M, computed on the short side of the matrix. Momentum doesn’t touch the bound, since any orthogonal update sits on the ball. It changes what’s being minimized: MM stands in for the gradient, lagged by about β/(1−β)\beta/(1-\beta) steps. It’s there for a reason. The polar factor is nonlinear, so the average of polar factors of noisy gradients isn’t the polar factor of the mean, and averaging first removes most of that bias. In the classical reviewer’s toy, with noise at twice the signal’s singular values, the signal direction keeps 64% of its size without momentum and 96% with it.

Two independent choices separate these optimizers: whether the geometry is per element or per matrix, and whether a running second moment moderates the step.

No second momentSecond moment in the denominator
Per elementSignum: sign⁡(m)\operatorname{sign}(m)Adam: m/vm/\sqrt{v}
Per matrixMuon: (MM⊤)−1/2M(MM^\top)^{-1/2}MShampoo, one-sided: L−1/2ML^{-1/2}M, with LL an average of ∇∇⊤\nabla\nabla^\top

Shampoo is an older optimizer that accumulates such a matrix preconditioner. Jordan et al. note that Muon with momentum off is an “accumulation-free” Shampoo. Muon’s lower memory comes from having no denominator, which Signum shares, and it gives up the noise term for the same reason. A comparison of AdamW and Muon changes both axes at once. Wen et al. run relatives of all four cells on one model: after tuning, the sign-based Lion stays close to AdamW (under 1.2×), while Muon, Soap and Kron give about 1.3× under 520M. That points at the matrix axis and not at the denominator, on models this small.

The same frame gives the element-wise side a comparison with a caveat. Sign descent is steepest descent inside a max-norm ball, where each element may move by at most η\eta, and Adam reduces to it when its moving averages are switched off (Bernstein and Newhouse). With the averages on, Adam is a “cousin”. Those two limits allow very different updates. Take a d×dd \times d update with element RMS rr, so that ∑iσi2=d2r2\sum_i \sigma_i^2 = d^2 r^2. With all of that in one singular value, as a sign update has when the gradient is rank one, it’s r⋅dr \cdot d. Spread evenly, as in an orthogonal update, each is rdr\sqrt{d}. So at equal element size the sign update can push the output d\sqrt{d} times harder along one direction: 85 times at d=7168d = 7168, V4-Pro’s width (computed), and s\sqrt{s} on a rectangular matrix with short side ss. That’s a worst case. We found no measurement of how close real gradients come, though Jordan reports from manual inspection that SGD and Adam updates on transformer matrices “are almost low-rank matrices”.

The other side is first-order progress. Equal element RMS means equal Frobenius norm. Take a gradient that is rank one with equal-size entries, plus a small full-rank remainder. Its cosine with its sign update is near 1, and with its orthogonal factor near 1/d1/\sqrt{d}, because the factor lifts the remainder’s directions to full size and spends most of its length where the gradient is small. On that gradient, at equal RMS, the sign update reduces the linearized loss about d\sqrt{d} times more. That needs a converged factor: with the remainder at a thousandth of the signal, five Newton-Schulz steps leave those directions small and the cosine is about 0.8 (computed, d=32d = 32). With no remainder at all, the factor is the sign update. Muon is the best step inside its own ball, and not per unit of weight movement. The case for it rests on what a loss does beyond first order and on noise, and neither the derivation nor these toys measure that.

Sign update (Signum): largest 32.00r, limit r·d = 32r; cosine with the momentum 0.97.

Orthogonal factor: largest 5.66r, limit r·√d = 5.66r; cosine with the momentum 0.37, or 0.33 after 5 Newton-Schulz steps with Jordan's coefficients.

Figure 2.A toy 32 × 32 matrix. As the momentum approaches rank one, its sign update has nearly all its strength in one singular direction, near 32 times the element size, while the orthogonal update stays flat at 5.7 times at every alignment. The readouts show the other side: near the slider's top the sign update's cosine with the momentum is near 1 and the exact orthogonal factor's near 0.18, and at the default about 0.97 and 0.37. Five Newton-Schulz steps stay nearer the sign update when the remainder is very small.

Computing UV⊤UV^\top by SVD at every step would be too slow, so Muon uses Newton-Schulz iteration: divide the momentum by its Frobenius norm so all singular values lie below 1, then repeatedly apply an odd polynomial. Since XX⊤X=UΣ3V⊤XX^\top X = U\Sigma^3V^\top, such a polynomial acts on each singular value and leaves UU and VV alone, using only matrix products. One training step is:

M ← βM + (1−β)∇                          momentum, often in Nesterov form
X ← M / ‖M‖_F
repeat K times:  X ← aX + b·XXᵀX + c·(XXᵀ)²X
W ← W − ηγX − ηλW                         γ is a scale, below; λ weight decay

Jordan et al. use K=5K = 5 and (a,b,c)=(3.4445,−4.7750,2.0315)(a, b, c) = (3.4445, -4.7750, 2.0315).

A flat spectrum after Frobenius normalization starts at 1/√s, 0.018 for s = 3072.

Figure 3.Each line is one singular value. Jordan's coefficients reach the band in four steps from 0.01. At 1 they sum to 0.701, so they oscillate at about the band's edges and never settle: Jordan accepted this after finding that an error up to about 0.3 doesn't harm the loss curve. The baseline coefficients settle exactly and slowly. DeepSeek V4 runs Jordan's for 8 steps and then the baseline for 2.

Labs use KK from 5 (GLM-4.5, Z.ai) to 16 (NVIDIA’s study), and Step 3.5, Qwen3.8 and NVIDIA use Polar Express, whose coefficients change per step by solving a minimax problem. The sources disagree on whether accuracy helps. Moonlight saw no gain from 10 steps over 5, and Amsel et al. found no gain beyond 6 or from an exact SVD on GPT-2 Small. Qwen found that 8 steps reduced gradient-norm spikes in a stress test, and NVIDIA reports slightly lower loss from exact polar factors on a 30B model. Number formats differ too. DeepSeek reports the iteration is stable in BF16 under its coefficients. StepFun saw rare unrecoverable spikes, simulations pointed at accumulated error in BF16 Polar Express at K=6K = 6, and it moved the iteration to float16.

The step has nearly the same size however small the gradient is, so without weight decay nothing limits the weights’ growth. The provable bound is linear in steps. Pethick et al. observe growth as the square root of steps on CIFAR-10, for a close relative of Muon. Moonlight saw weights and layer outputs grow past BF16’s high-precision range when scaling up. With decay, the update is a Frank-Wolfe step: WW becomes a convex combination of its old value and −(γ/λ)X-(\gamma/\lambda)X, so decay sets a ball of radius γ/λ\gamma/\lambda that weights starting inside can’t leave (Pethick et al.), give or take the iteration’s overshoot. With RMS matching and λ=0.1\lambda = 0.1 that ball is far larger than real weights: element RMS at most 2, where Moonlight’s small-scale runs show under 0.1 early in training (computed). So there decay acts as a pull toward zero and isn’t a binding limit. Kolibri’s fixed decay gives a much smaller ball, about 4 in spectral norm at its peak rate, against about 2 at initialization (computed).

Whether it helps depends on the horizon. On an 800M model trained well past its optimal token budget, vanilla Muon led by 0.023 in validation loss at iteration 24,000, and decayed Muon by 0.017 at 66,000 (Moonlight). In Lion et al.’s 1B and 2.7B runs, Muon’s best result had no decay at all, though decay helped at 124M. Their Muown (used by Xiaomi from mid-training) treats each row’s magnitude as its own variable, a form of weight normalization, updated by Adam, and applies Muon to the direction. They report no spectral drift and lower perplexity than Muon in dense runs up to 2.7B. It needs a non-zero start for any matrix that would be zero-initialized, and its authors leave mixture-of-experts models open, which is where Xiaomi uses it.

What Muon costs to run

Jordan et al. count 2(2ℓs2+s3)2(2\ell s^2 + s^3) FLOPs per iteration on a matrix with long side ℓ\ell and short side ss, at most 6ℓs26\ell s^2. They count a forward and backward pass through the same layer at 6ℓsB6\ell s B for BB tokens. So over KK iterations the overhead is at most Ks/BKs/B, which, as Essential AI points out, shrinks with batch.

In a mixture-of-experts (MoE) layer, a router sends each token to kk of NN expert sub-networks, so an expert sees only about B⋅k/NB \cdot k/N tokens. Take V4-Pro at K=10K = 10. A dense matrix at its width of 7168 and the peak batch of 94.4M tokens gives 0.08%. An expert with hidden size 3072, active for 6 of 384 slots, sees 1.5M tokens and gives 2.1% (both computed). That’s the favourable end: on NVIDIA’s 72B model, with K=16K = 16 and an expert side of 2688, the same formula gives 7.3% at the 50M tokens its Muon ran at, and 14.7% at 25M (computed).

The bound also leaves out what a training system pays for: communication, waiting for the slowest rank, many small kernels, and number formats that differ between the forward pass and the iteration. Reported times come from other systems. Moonlight says usually 1% to 3% of forward-backward time at K=5K = 5. Poolside reports less than 1% of the step on Laguna M.1, with communication overlapped with computation. DeepSeek prints no timing for V4-Pro.

Large training systems split each matrix across GPUs, and Muon’s update needs the whole matrix.

Figure 4.AdamW's state and arithmetic are per element, so a matrix can be cut anywhere. Muon's update depends on each whole matrix, so the system gathers it, computes, and sends the result back. This shows one owner rank per matrix, the design Poolside and Qwen describe. Moonlight gathers on every rank and discards what it doesn't own, and NVIDIA also describes all-reducing inside each iteration.

Six labs describe a fix. DeepSeek assigns whole dense matrices to ranks with a knapsack packer, and reports under 10% memory overhead from padding on those when each rank manages at most five matrices. Moonshot’s K3 fetches only the shards each rank owns over peer-to-peer links. Z.ai restricts its all-gather the same way. Qwen’s Canzona balances estimated iteration FLOPs across ranks, and captures the many small kernels in a GPU command graph. Poolside gathers to one owner per parameter. StepFun reports about 5% less iteration time than its own naive all-reduce path, for under 4 GB of extra memory. Moonlight’s earlier design calculates communication at between 1 and 1.25 times distributed AdamW’s, which assumes the gather is in 16 bits.

The work brings new ways to fail. Poolside’s hash checks caught non-determinism in the iteration, “whose extra norm reductions and matmuls offer more opportunities for non-determinism than AdamW”, and it pinned the relevant kernels to deterministic algorithms.

What the measurements say

Who ran itScale, tokensComparisonMuon vs AdamW
Pethick et al. (EPFL)124M; batch sweep with step size tuned per optimizer and batchValidation lossScion “better at maintaining a low validation loss” than Muon and Adam as batch grows
Wen et al. (Stanford)130M to 520M swept per optimizer; 1.2B by a fitted law; denseTokens to a loss1.4× at 0.1B, 1.1× at 1.2B; Nesterov AdamW also about 1.1×; at 1.2B and 8× budget, mean of 10 benchmarks 66.98 vs 67.15
Moonshot (Moonlight)399M to 1.5B, batch 0.8M to 2.1M tokensFLOPs along a fitted law“0.519x FLOPs”, the source of “2×”
Jordan et al.1.5B, short run, both tunedTime to GPT-2 XL on HellaSwag10 vs 13.3 hours on eight H100s
Moonshot (Moonlight)2.24B active, 15.29B total; 1.2T of 5.7T tokens, before the learning rate decaysSame recipeHumanEval 37.2 vs 29.3, MATH 19.8 vs 16.1, MMLU 60.4 vs 60.2, BBH 43.2 vs 45.3
Pethick et al.3B GPT, far below a full budget, no seeds printedValidation loss2.909 vs 3.024
Essential AI100M to 4B; ratio plots to 10M tokens, a 500M sweep to 16M; denseTokens to a training loss10% to 15% fewer in its conclusion, which fits its plots at the lowest target losses; much more at high targets and large batches
NVIDIA3B active of 30B at 1×; 8B active of 72B at 2×; 1T to 3T tokensMuon at 1× to 3× of 25M tokens vs AdamW at 1×72B: HumanEval 65.79 vs 57.62, Math 500 75.60 vs 80.20. 30B at 1×: Math 500 75.50 vs 71.50, HumanEval 62.56 vs 63.26
Qwen28-layer 25B test model, Qwen3.5 structure, constant raised rateLoss spikes per 10k steps0.2 vs 4.3 at twice the rate it calls optimal

Higher benchmark scores are better. None of these comparisons runs as long as a shipped model: the longest is 3T tokens, against 15.5T for K2 and 33T for V4-Pro. MoE rows also give size two ways, and tokens per parameter differ by 7× to 10× depending on which you count, and by about 32× on V4-Pro (computed).

The 2× and the 1.1× are both FLOP ratios, since Essential AI notes that a token ratio is a FLOP ratio at fixed model size. They still measure different things. Wen et al. and Essential AI hold a model fixed and count tokens to a loss. Moonlight fits loss against compute as L=aC−bL = a C^{-b}, with CC about 6 × parameters × tokens, along a path where model size and tokens grow together. Its batches are small, so its 2× is not a large-batch result.

Figure 5.Moonlight's two fitted curves (Table 3), with C in PFLOP/s-days, a unit we inferred. At the same compute the gap is about 4%. Since loss falls so slowly with compute, matching it takes about twice the FLOPs. The solid part is the range the fit was made on, and the printed 0.519× falls beyond it. The y axis is truncated.

A flat curve makes the ratio sensitive: R≈(1+g)1/0.053R \approx (1+g)^{1/0.053} for a gap gg, so each percentage point of gap moves the ratio by about 20% (computed). On this scale 1.1× is a gap of 0.5% and 2× is one of 4%, a difference of about 0.08 nats, with no interval printed on any fit. The baselines differ as well. Moonlight grid-searched AdamW’s learning rate and batch size and gave Muon the same values, so tuning Muon separately could only have helped Muon. What’s open is how strong the AdamW baseline was. Wen et al. sweep more settings per optimizer and write, of the literature generally, that “many reported speedups of 2× simply reflect a weak baseline”. Neither paper isolates how much of the gap this explains. Moonlight fits at compute-optimal budgets, 22 to 26 tokens per parameter by its Table 2, which Wen et al. call 1× Chinchilla (about 20), and Wen et al. state their 1.1× at 8×.

At 1.2B Wen et al.’s Nesterov AdamW reaches the same 1.1×, so in that range the speedup doesn’t need a matrix optimizer at all. They also predict from a fit that at 7B and 1× Chinchilla Muon would end with a higher loss than AdamW, by 0.09% on their printed coefficients (computed), six times past their largest model. That’s effectively a tie. Above 1.2B the pre-training comparisons we found have Muon ahead: Jordan et al. on time to a benchmark (both tuned, short run), and Pethick et al., Essential AI, Moonlight and NVIDIA on loss. Their benchmarks are less uniform. Moonlight and NVIDIA didn’t tune each optimizer separately and Essential AI doesn’t say. No study we read has both per-optimizer tuning and that scale.

Large batches are where Muon’s case is strongest. Essential AI sweeps batch from 128K to 16M tokens on a 500M model, and finds Muon “more effective than AdamW in retaining data efficiency at large batch sizes”. NVIDIA reports AdamW not benefiting from 50M-token batches on its model, without re-tuning AdamW’s betas or ϵ\epsilon. Its Muon at 2× runs against AdamW at 1× over the same tokens, so Muon gets half the updates and its loss result is the stronger for it. AdamW at 2× has its own chart, without Muon on it. NVIDIA hypothesizes that with less noise AdamW’s second-moment estimate becomes “poorly calibrated”. We found no study that isolates this. In the toy above a re-tuned learning rate would cancel Adam’s rising RMS, so that explanation can only separate runs that keep one rate. The one protocol that tunes the step size for each batch and optimizer is Pethick et al.’s, at 124M.

Sources differ on whether Muon needs a batch ramp. On a 10.8B test model, Qwen’s run ramping to 25.2M tokens ended within run-to-run variance of starting there and cost 18.8% more steps, so it doesn’t ramp in production. No AdamW ramp was run. NVIDIA ramps both optimizers for its larger batches and credits the ramp for Muon’s lead. DeepSeek V4, Z.ai and StepFun schedule their batches too, while DeepSeek V4.1, like Qwen, runs its batch fixed from the start.

NVIDIA’s MoE argument cuts in an unexpected direction. Experts “remain in a lower-effective-batch regime”, and raising the global batch “primarily stresses” the dense parts. On V4-Pro an expert sees 1.5M tokens against 94.4M for dense parts. Experts also hold most of the parameters and have the worst overhead bound. So the large-batch evidence bears mainly on a small share of the matrices (our reading). We found no study that isolates this, for instance by running Muon on dense parts alone.

KL-SOAP goes further along the matrix axis: a Shampoo-style method with Kronecker factors and an eigenbasis, at a “much larger” memory cost. NVIDIA finds it has a “slight edge” over Muon and recommends it where memory isn’t limiting, after fixing a stale-basis instability. Wen et al. find Muon “outperformed by Kron and Soap” at 8× Chinchilla or more.

On stability, DeepSeek V4 names “improved training stability” with no measurement, and the same report describes loss spikes tied to MoE outliers and fixes them by other means. On the 25B test model at 4× the rate it calls optimal, Qwen’s AdamW spiked 183 times per 10k steps and hit the clipping threshold on 213 of 19,932 steps, while Muon never hit it (its spike rate there isn’t printed). The same report toggles a gating layer under AdamW at another rate and takes spikes from 32.0 to 3.2 per 10k. So architecture moves that number by the same order, 10× against about 21× for the optimizer at 2× (computed), and those figures aren’t on one ladder, since the report’s figures imply different optimal rates.

Moonshot’s K2 report says exploding attention logits occur “more frequently with Muon but less with AdamW”. On a 53B test model with plain Muon they “quickly exceed a magnitude of 1000”, and K2 was trained with an added clip on queries and keys, which it calls MuonClip. GLM-5 and Qwen3.8 report no such clip, the first crediting its head split. Moonlight’s appendix says AdamW “demonstrated healthier behavior” on that metric. AdamW runs diverge too: NVIDIA traces one Nemotron 3 Ultra divergence to a precision change in the output layer, and calls a second undetermined. None of the reports we read describes abandoning Muon or calls an AdamW run a mistake, and published recipes describe what shipped.

Parameters that aren’t linear maps

Bernstein’s derivation assumes dense inputs and a matrix mapping activations to activations, and he says that “separates the Linear layer from, say, an Embedding layer that takes one-hot inputs”. Several kinds of parameter fall outside it, or stay off Muon anyway.

  • Scalars and vectors have nothing to orthogonalize: norm scales, biases, the per-head decay and beta projections in Qwen3.8’s Gated DeltaNet layers.
  • An embedding table takes one token at a time. The inputs that occur are one-hot vectors, and on them the output change is one token’s vector. The spectral norm also limits outputs on mixtures of tokens, which never occur, so it constrains the wrong set (our reading of Bernstein’s exclusion).
  • The LM head has a dense input, so the theory doesn’t exclude it. Jordan still keeps it on AdamW, and says this “does not seem to follow from the theory, and is instead driven by empirics”.
  • The router’s outputs are scores for separate experts. Qwen saw Muon destabilize it early in training, with no gain later, and offers the independence of those scores as “one possible explanation”.
  • Qwen’s low-rank projections do better on AdamW, which it attributes to “their very elongated shape”.
  • NVIDIA saw Mamba’s Conv1D filters give about 0.1% higher loss under Muon, and “sometimes” go to NaN. Its reason is that they’re shared and re-applied at every position.
GroupKolibriDeepSeek V4DeepSeek V4.1GLM-4.5Qwen3.8
Attention Q/K/V, expertsMuonMuon, by exclusionMuonMuon, experts by exclusionMuon
Token embeddingAdamAdamWSinkhorn-momentumnot Muon, unnamedAdamW
LM headAdamWAdamWSinkhorn-momentumMuon, by exclusionAdamW
MoE routerAdamWMuon, by exclusionMuon, by exclusionAdamW
Norm scalesAdamAdamWAdamWnot Muon, unnamed
Gates, scaling factorsAdamW (mHC)AdamWAdamW (output gates)
N-gram tableSinkhorn-momentumAdam, no decay

“By exclusion” means the report lists what isn’t on Muon and says everything else is. Kolibri’s “Adam” groups carry no weight decay, which makes them the same update as AdamW. MiMo-V2.6 keeps embeddings, head and router on AdamW from mid-training. So labs split the parameters differently, and they put the router and head on different optimizers.

Labs that keep AdamW run the two optimizers together in one of two ways, and a third approach removes it from most of these groups.

The first matches update sizes. Moonlight’s Lemma 1: a full-rank orthogonal update on an ℓ×s\ell \times s matrix has element RMS 1/ℓ\sqrt{1/\ell}, so large matrices get small updates. Moonlight multiplies by γ=0.2ℓ\gamma = 0.2\sqrt{\ell} to fix RMS at 0.2, in AdamW’s observed range, so that one learning rate and one decay can serve both. DeepSeek uses 0.18. Poolside, in preliminary experiments, ran Muon without this rescaling and with standard decay coefficients, and its experts collapsed from about 450B tokens, spreading layer by layer through the first three MoE layers until divergence. That run changed two things together and had no AdamW arm. Poolside’s explanation is that rescaling lets Muon run “at AdamW-scale LRs”, removing “the order-of-magnitude mismatch in effective WD”. The cost is that RMS matching gives up the output-change limit that motivates Muon, and it ties 0.2 to AdamW’s betas, as the Adam figure shows. Bernstein writes of his own factor, “I’d actually expect the one here to scale better”.

The second gives each group its own rate and makes Adam scale-free. Kolibri (Aleph Alpha) uses Bernstein’s factor and separate rates per group: 1e-3 for Muon, 2.5e-5 for router and head, 2e-3 for embeddings and norms. It sets ϵ\epsilon = 1e-15 so that Adam’s updates on small-gradient parameters “stay independent of the gradient scale, as Muon updates are by construction”. The cost is three rates and a per-step decay to tune and carry through every stage.

The third drops Adam and normalizes these parameters directly. If the argument for Muon is to choose a norm that suits the layer, an embedding has its own best norm. For one-hot input the natural limit is on each token’s vector separately, and normalizing each vector is exactly the best step inside that limit. This is a worst case over tokens: it treats a token seen once like one seen a million times, and its direction then rests on few samples. Adam already gives a rare token a similar size on the steps it appears, though its average per step falls with the root of its frequency. What normalization gains is direction: on a row with Gaussian entries it beats the sign update by about 1.25× in linear decrease at the same element RMS, and by more for heavier tails (our computation). An orthogonal factor doesn’t equalize tokens at all: row ii of UV⊤UV^\top has squared norm ∥Ui⋅∥2\|U_{i\cdot}\|^2, its leverage score, which follows frequency.

Rows: tokens, most frequent first. Bars: each token's update size.

Figure 6.A toy embedding update with tokens as rows. In raw momentum, common tokens dominate. The sign panel already has equal row sizes. The orthogonal factor leaves row sizes uneven, following frequency. Normalizing rows gives each token an equal step along its own direction, and alternating with columns (V4.1's method) balances features as well. V4.1 normalizes momentum, so a token absent from a batch still has a fading row, and normalizing restores it to full size: a row of average size keeps moving for about 135 steps, until it falls below V4.1's mask.

Scion (Pethick et al.) works out these choices per layer as conditional gradient, the family of Frank-Wolfe. For one-hot input it recommends normalizing each token’s vector, for the last layer the sign, and with tied weights the sign on both ends. Its reported GPT results use the tied form, so they test a norm-based optimizer without Adam, and not per-token normalization. Orvieto and Gower note that Scion’s architecture has no trainable gain in its normalization layers, and that “experiments in the usual setup are needed”. Its memory claim, one set of gradients in half precision and no separate momentum buffer, should be weighed against production runs that keep gradient accumulation in FP32 for stability.

DeepSeek V4.1 is the production case, and it goes past this frame. It applies Nesterov momentum and then Sinkhorn balancing to embeddings, head and Engram tables (its name for n-gram tables): alternately normalize rows (one token) and columns (one feature), 11 passes, then scale by 0.18 to match Adam’s size. The column passes mean the update is no longer the best step inside any fixed ball. It is a diagonal scaling of the momentum on both sides, so read it as a preconditioner, a diagonal relative of Shampoo. V4.1 uses no weight decay on these groups, and says this “requires only a momentum buffer while empirically outperforming Adam”. It prints no number, and it’s the only lab we found stating this. SinkGD (Scetbon et al.), which V4.1 cites, is a different setup: stateless, on linear layers, with Adam kept on embeddings, norms and head. Norm scales and biases stay on AdamW at V4.1, so no shipped model we read states that it runs without Adam.

After pre-training

A recipe continues after pre-training with mid-training, then supervised fine-tuning (SFT) on prompt-and-response examples, and often reinforcement learning (RL) against a reward (01). Jordan’s post asked whether Muon “works only for pretraining”. Moonlight gave the first answer, fine-tuning checkpoints taken partway through pre-training, at 1.2T of 5.7T tokens, for two epochs on an open SFT mixture:

Pre-train → SFTMMLUHumanEvalMBPPGSM8K
Muon → Muon55.757.355.668.0
AdamW → Muon55.353.755.562.1
Muon → AdamW50.252.455.264.9
AdamW → AdamW52.053.155.264.6

Muon through both stages is best on every column, and no row has seeds or error bars. On a Qwen2.5-7B base, Moonlight’s Adam-SFT beats Muon-SFT on all four benchmarks, GSM8K 89.8 to 85.8, a result it calls “on par”. It concludes that Muon is better applied in pre-training than in SFT, and names an “optimizer mismatch” as an open problem: the best SFT optimizer depending on which optimizer pre-trained the checkpoint. Later studies disagree about how large that effect is, and they use different metrics.

Figure 7.Each panel holds one result, with its own y axis, so slopes don't compare across panels. If each checkpoint does best with its own optimizer, the lines cross: Qu et al.'s and Moonlight's GSM8K do. If one SFT optimizer is better on both checkpoints, they don't: Moonlight's MMLU and Zhao et al.'s 1×. Zhao et al.'s 8× crosses by 0.1, within its ±0.4, though Muon SFT there helps only the Muon checkpoint. Scales differ (561M to 2.24B active), and so do the SFT data and tuning.

Qu et al. (ICML 2026) find a symmetric mismatch on a 561M model pre-trained from scratch, then test why. They link it to implicit bias, the kind of solution each optimizer tends toward: for sign descent on a linear regression toy the minimum max-norm solution, and for Muon the minimum spectral-norm one. They offer this as a hypothesis, with theorems about the toy, started from zero, with no momentum and step sizes going to zero. Sign descent stands in for Adam, and no theorem covers a change of optimizer partway. They then show that the damage grows with update strength. Under LoRA (low-rank adapters that limit how much weights can change), the perplexity gap on that 561M model falls by 39% and 78%, and pooled across their T5, Llama and CLIP tasks by 0.72%. LoRA-Muon matches LoRA-Adam on Llama 2-7B, though full fine-tuning with Adam stays above both, a mean score of 54.92 against 52.15.

Zhao et al. (a NeurIPS 2026 workshop paper) find “little evidence of a meaningful optimizer mismatch penalty”. They select the learning rate per setting on held-out loss and report three trials. Final scores are “largely explained by the independent effects” of the two optimizers, with Muon SFT ahead on AdamW-pretrained open models in 4 of 5 settings up to Qwen3-14B, mostly by less than run-to-run variation. Their result on Qwen2.5-7B disagrees with Moonlight’s on the same base and the same dataset name. The two differ in tuning and in evaluation (GSM8K scores sit about 10 points apart), so we can’t say which explains it.

Li et al. use a different measure, a trade-off between learning the new task and keeping the old, on models up to 600M, and find that AdamW overtakes Muon after fine-tuning once the pre-training budget is large enough, and earlier for larger models. Bagai et al. report, in a workshop abstract (we couldn’t read the paper), that on Qwen3-4B Muon trails Adam until the learning rate of its auxiliary AdamW group is tuned, after which the two are “indistinguishable”.

These sources don’t settle it, and none tests a mixture-of-experts checkpoint at frontier scale. Across them, both of Muon’s rates need tuning at SFT, its own and that of the parameters still on Adam, and a comparison at one fixed schedule is unreliable.

Of the seven labs naming an optimizer both in pre-training and at a later stage, five keep one family: Aleph Alpha, Moonshot (K2), Poolside and StepFun on Muon in SFT, NVIDIA on Adam. Two switch. Mistral states Muon for RL, with no weight decay, and doesn’t state its mid-training or SFT optimizer. Xiaomi moves to Muown in mid-training “to prepare the model for subsequent large-batch RL”, because its experiments “showed diminishing optimization efficiency as the batch size increased in mixed-task RL”, with no number printed. It reports “no loss spike throughout the training process”. That statement covers spikes, and not the end quality the fine-tuning studies measure. Ruan et al., on Qwen2.5 models up to 3B on ALFWorld with five seeds per cell, tie much of Muon’s late gain in RL to a larger effective step: an RMS-matched control removes it, and tuned AdamW nearly matches at 3B on one objective. AdamW couldn’t take that step directly, since higher rates failed under the shared recipe, though they note joint tuning with regularization might change that. Fan et al. report the opposite outcome on single-turn RL: on Qwen3-1.7B and 4B, at one learning rate shared across optimizers over 40 to 80 steps, Muon’s accuracy falls to near zero on MATH and GSM8K while AdamW improves. They blame whitening, which lifts noise directions in low-signal gradients to full size, and propose a filtered replacement. They print no seeds or rate sweep, and Ruan et al. read that failure as rate dependence. Six labs state Muon or a variant in RL and report no collapse. So the labs’ RL runs and these two studies don’t yet fit one explanation.

Reading a recipe

Learning rates compare only under one convention. RMS-matched multiplies by γ=cℓ\gamma = c\sqrt{\ell}, with cc about 0.2: Moonlight, K2, DeepSeek (0.18), Laguna, Qwen3.8, GLM-4.5. Bernstein’s multiplies by dout/din\sqrt{d_{\text{out}}/d_{\text{in}}}: Kolibri, and probably Mistral in RL, which says “spectral scaling” with no formula. Jordan’s original form, max⁡(1,dout/din)\sqrt{\max(1, d_{\text{out}}/d_{\text{in}})}, survives in smaller studies. On a square d×dd \times d matrix RMS matching and Bernstein’s differ by 0.2d0.2\sqrt{d} at the same rate, about 17 at d=7168d = 7168 (computed).

× learning rateelement RMSlargest singular valueoutput-change limit
raw orthogonal1.18e-51.00e-31.00e-3
RMS-matched 0.2·√ℓ2.00e-41.69e-21.69e-2
RMS-matched 0.18·√ℓ1.80e-41.52e-21.52e-2
Bernstein √(d_out/d_in)1.18e-51.00e-31.00e-3
Figure 8.Enter a matrix shape and a learning rate. RMS matching fixes the element size whatever the shape, which leaves the largest singular value proportional to the root of the long side, and the output-change limit larger still on matrices that shrink their input. Bernstein's factor fixes the output-change limit at 1 and leaves element size and singular value to vary. So 1e-3 at Kolibri and 2.0e-4 at DeepSeek V4-Pro aren't on one scale.

Weight decay compares only under a stated formula. Where a formula is printed (K2, DeepSeek V4, and the Moonlight and NVIDIA papers), decay is ηλ\eta\lambda per step, and six of the 11 labs state 0.1. As frameworks implement it, AdamW’s decay also scales with the learning rate, and Essential AI calls that “coupled”, a different sense from “decoupled” above. Kolibri decays by 2−122^{-12} per step, independent of the learning rate. Inkling (Thinking Machines) “coupled the weight decay strength to the square of the learning rate”, pointing to Kosson et al. and Defazio without giving a formula. In RL, Aleph Alpha, Mistral and Xiaomi use none, and StepFun keeps 0.1.

Many Muon setups treat Q, K and V as separate matrices, as Jordan advised, and several split further by head: GLM-5’s Muon Split, K3’s Per-Head Muon, V4.1’s head-wise Muon, and Qwen3.8’s split fused parameters. Qwen’s argument is that one orthogonalization over a concatenated matrix mixes singular directions across unrelated sub-blocks. Splitting limits each head’s output and gives up the limit on the layer: the concatenated update’s largest singular value can rise by up to h\sqrt{h} at the same element RMS, for hh heads (classical reviewer’s computation). K3 says the iteration gets slightly cheaper, and Qwen says the step fragments into many small kernels. Only GLM-5 prints numbers for a head split: on its table, one attention variant with Muon Split beats its comparison on four benchmarks and trails on three, which it calls matching. NVIDIA tested the earlier step, splitting a fused Q, K and V tensor, on a 30B model. Loss was lower early, and “the losses at the end of the full 1T-token horizon are identical”. That is loss alone, and it doesn’t test splitting by head.

The options side by side

OptimizerState per parameterStep costShardingEvidenceLabs in pre-training
Signum1 bufferelement-wiseanywhereabout a third more budget than Adam (160M)
AdamW2 bufferselement-wiseanywherethe baseline; a noise-scaled step3 of 11
Muon1 bufferKK iterations per matrixwhole matrixsee the table above8 of 11
Muown1 buffer plus four short vectors per layerabout 1.5% over Muon, nothing when shardedas Muonlower perplexity than Muon to 2.7B (its authors)Xiaomi, from mid-training
Momentum + Sinkhorn1 bufferrow and column norms, KK passesnot stated“outperforming Adam”, no numberDeepSeek V4.1, on embeddings and head
Scionno buffer beyond the gradientas Muon, sign elsewhereas Muon3B without Adam, no trainable norm gain
KL-SOAP“much larger”matrix products, eigenbasis updatesmatrix-based“slight edge” over Muon on a 30B model

Where the models in the map sit

LabLatest model stating itPre-trainingLater stages, where stated
Aleph AlphaKolibri-1Muon + Adam + AdamWsame groups in SFT; RL moves head and router to Adam
MoonshotK3per-head MuonK2: Muon in SFT; K2.5: MuonClip in RL
DeepSeekV4.1 (V3 was AdamW)Muon + AdamW + Sinkhorn-momentum
Z.aiGLM-5Muon
PoolsideLagunaMuon + AdamWMuon “across all training stages”
Qwen3.8 (3.5’s AdamW is implied by the 3.8 report)Muon + AdamW + Adam
Thinking MachinesInklingMuon + Adam
StepFunStep 3.5 FlashMuonMuon in SFT and RL
MistralLarge 4AdamWMuon in RL; others not stated
XiaomiMiMo-V2.6AdamWMuown on hidden matrices from mid-training
NVIDIANemotron 3 Nano, Super (Ultra inherits)AdamWAdam in 3.5 SFT, AdamW in quantization-aware distillation

So 8 of the 11 labs that state a pre-training optimizer use Muon. StepFun’s and NVIDIA’s statements are in reports outside our corpus; on corpus documents alone the count is 7 of 9. Liquid uses AdamW in SFT and doesn’t state its pre-training optimizer. OpenAI, Google, MiniMax and Tencent don’t state one in the documents we read. NVIDIA’s Muon paper is a research study and doesn’t establish what any shipped Nemotron used.

Choosing

This is our judgment, and the sources don’t settle it. For a new large model trained from scratch at large batches, on a system you can change, we would start with Muon on linear maps. In the comparisons above 1.2B it comes out ahead on pre-training loss or time, its large-batch evidence comes from two groups’ sweeps (Essential AI, NVIDIA), and most labs chose it, though few print a comparison of their own. Against it, AdamW leads on math at NVIDIA’s 72B and on BBH at Moonlight’s 2.24B active, Wen et al. find 1.1× and no downstream gain at 1.2B with Nesterov AdamW level, and Li et al. find the ordering reverses after fine-tuning at up to 600M.

Don’t build a budget on 2×. It comes from one fitted law, at batches around 2M tokens. Don’t build one on 1.1× either: that stops at 1.2B and dense models. No study measures the gain at your scale, so if the engineering is what’s being decided, run your own comparison at your batch, with each optimizer tuned. Keep AdamW or Adam for scalars and norms, and test the head, router and embeddings on your own architecture. If you have no reason to tune each group, RMS matching is the convention with the most users.

AdamW remains defensible at small batches, on a training system you can’t change, or wherever the budget doesn’t pay for the engineering. Three labs ship it in pre-training, and Wen et al. report that a carefully tuned baseline leaves 1.1× at 1.2B and that Nesterov AdamW reaches about the same. Nesterov and equal betas are the cheap things to try first. Raising β1\beta_1 to reach equal betas shrinks the step, so the learning rate needs re-tuning.

Moving a checkpoint to Muon later has one stated reason, large-batch RL at Xiaomi, with no number printed. Ruan et al. tie much of the measured gain to a larger step that AdamW couldn’t sustain, and Fan et al. show Muon collapsing outright at a rate shared with AdamW, so screen the rate before committing. Plain SFT is an open question. Tune every group’s learning rate for whichever optimizer you pick, and treat a comparison at one fixed schedule as unreliable.

Easy to get wrong

  • “Muon is 2× better.” 2× is FLOPs along one fitted law, where model size and tokens grow together, at batches around 2M tokens. Holding the model fixed, Essential AI’s summary has 10% to 15% fewer tokens, matching its plots at the lowest target losses, and Wen et al. have 1.1× at 1.2B.
  • “NVIDIA shows AdamW failing at Muon’s batch.” Its Muon at 2× runs against AdamW at 1× over the same tokens. That makes the loss result stronger. AdamW at 2× did run, on a separate chart with no gain, but its betas and ϵ\epsilon weren’t re-tuned, and AdamW still leads on math at 72B.
  • “Muon halves training memory.” It halves optimizer state, which on FP32 weights and gradients is about a quarter of the per-parameter total. Gathering whole matrices raises the peak on a rank.
  • “8 of 11 labs tested it and chose Muon.” Two of the eight print a comparison of their own. The other 6 cite others, state an observation or give no reason, and 3 labs ship AdamW.
  • “Muon fixed stability.” Qwen’s gate has an effect of the same order under AdamW, and K2 needed a clip because of Muon, on attention logits where it and Moonlight report AdamW better.
  • “A Muon learning rate of 1e-3 is higher than one of 2e-4.” On a square 7168 matrix, Kolibri’s convention gives an output-change limit of 1e-3, and DeepSeek’s gives about 3e-3 (computed).

Check yourself

A gradient on a 1024 × 1024 matrix is rank one with equal-size entries, plus a tiny full-rank remainder. At equal element RMS, how do its sign update and its exact orthogonal factor compare on largest singular value and on first-order loss decrease?

The sign update has largest singular value 1024r1024r and the orthogonal factor 32r32r, so the sign update can move an output 32 times more. Its cosine with the gradient is 1 against 1/321/32, so it also reduces the linearized loss about 32 times more. Muon gives up first-order progress to bound output change. With no remainder at all, the two are the same matrix.

A 400B model trains at FP32. How much optimizer state does it hold under AdamW and under Muon, ignoring the groups Muon doesn’t cover?

3.2 TB and 1.6 TB: 400B × 4 bytes × 2, and × 1. Muon’s parameters that stay on Adam bring some of that back.

With 5 Newton-Schulz iterations, a 10M-token batch and 8 of 256 experts active, what does Jordan’s bound give for a dense matrix with short side 2048 and for an expert with short side 2048? Why do they differ?

0.10% and 3.3%: 5×2048/10M5 \times 2048 / 10\text{M}, and the same over 10M×8/25610\text{M} \times 8/256 = 312,500 tokens. An expert costs as much to orthogonalize but sees far fewer tokens, so its forward and backward pass is smaller next to it.

Why doesn’t a table of Muon → Muon beating every other row show that a mismatch exists?

It could also mean Muon is better in each stage independently. A mismatch means the best SFT optimizer depends on the pre-training optimizer, so you need to compare the SFT effect on each checkpoint. On Moonlight’s GSM8K the effect of Muon SFT flips sign (+3.1 and −2.5), and on its MMLU it doesn’t (+5.5 and +3.3).

In the map

Methods

Taxonomy nodes

Further reading

↑