DeepSeek trained V3 with AdamW and V4-Pro with Muon. Mistral pre-trained Large 4 on AdamW, Xiaomi did the same for MiMo-V2.6, and NVIDIA for Nemotron 3 Nano and Super. Of the 11 labs we found stating a pre-training optimizer, 8 use Muon at their latest model that says. Two of those 8 print a comparison they ran themselves: Moonshot, and on stability Qwen. The rest cite others, report an observation without a number, or give no reason.
The switch has a price. Six of the eight describe something they built so that Muon could run on their training systems. Those that say which weights use it name the linear maps, so a second optimizer or a second rule stays in the run for the rest. Moonshot’s Moonlight paper reports reaching the same loss with about half the compute of AdamW. A Stanford benchmark, sweeping each optimizer up to about 0.5B parameters and extending to 1.2B by a fitted law, found a gain of 1.1× and shrinking.
This article explains what each optimizer does to a gradient and why that makes one of them expensive in a way the other isn’t. It then reads the evidence on pre-training, on the parameters Muon can’t handle, and on what happens when a Muon-trained checkpoint is fine-tuned.
Training recipes aren’t in the configs or modeling code we check, so everything here comes from documents: first the labs’ technical reports, then method papers, each named where it’s used. A blank in a table means we searched and didn’t find it, which says nothing about whether the lab does it. Tokens are pre-training tokens seen, and batch sizes are global, unless a sentence says otherwise.
What an optimizer costs
An optimizer turns a gradient into a weight update. At frontier scale that has a cost in space, in compute and communication, and in convergence.
Plain momentum keeps one running average of the gradient per parameter. Adam keeps two buffers per parameter, and Muon one. On V4-Pro’s 1.6T parameters, at FP32, that’s 12.8 TB against 6.4 TB (computed: 1.6T × 4 bytes × 2 or × 1). The dtype is our assumption: DeepSeek’s V3 report tracks its AdamW moments in BF16, a 16-bit format, which would halve both, and the V4 report doesn’t say.
The 2:1 ratio holds for the state alone. Counting FP32 weights and gradients as well, the per-parameter total falls by about a quarter (computed: 16 bytes to 12). On a large cluster the state is spread thin, and Moonshot’s K2 report calls its per-device footprint “negligible”. No lab names memory as a reason for choosing Muon in general. It motivated one narrower choice: DeepSeek V4.1 adds 196B parameters of n-gram tables and says that applying Adam to them “substantially increases the optimizer-state memory footprint”. Qwen3.8 has a smaller n-gram table and keeps it on Adam.
AdamW touches each element on its own, and NVIDIA calls it “infinitely shardable”: any GPU can update any slice of any matrix. Muon’s update depends on each whole matrix, so that cost needs Muon’s definition first, and comes after it.
Convergence, how far the loss falls per token, is why a lab would accept the second cost. It gets its own section after the mechanisms.
AdamW: one step size per weight
Adam divides each coordinate’s momentum by the root of an average of its squared gradients. With gradient at one coordinate:
This is a diagonal preconditioner: each weight gets its own scale, and weights never influence each other’s scale. Orvieto and Gower give an exact reading, building on Balles and Hennig. When and is ignored, is an online estimate of the gradient’s mean, and is an online estimate of its variance . Since , the step becomes
So at equal betas a coordinate moves by at most , and by less when its gradient is noisy. (At the common (0.9, 0.95) a run of same-sign gradients can briefly pass that bound.) Orvieto and Gower call this steepest descent inside a trust region set by the signal-to-noise ratio. The name is loose. A classical trust region tests each step and can reject it, while here the step always lands on the boundary, so the content is in the radius, which shrinks as noise grows.
The noise term matters in practice. Keeping only the sign of the momentum, the optimizer known as Signum, needed about a third more budget than Adam for the same loss on their 160M model, even after extensive tuning (computed from their “3/4 of the budget”). They recommend as a default, tested up to 410M parameters. Among the labs, only Mistral states equal betas in pre-training, (0.975, 0.975), and reports “moderate gains”. Others that state betas, such as DeepSeek on its AdamW groups, Aleph Alpha on Kolibri and NVIDIA, use (0.9, 0.95).
Moonlight observes that AdamW’s update RMS (the root mean square of its elements, per unit learning rate) sits around 0.2 to 0.4. Kosson et al. and NVIDIA give a factor of , with , which comes from averaging: an average of independent noise has variance . Combining this with the formula above gives a toy (our construction). For independent gradients with mean , variance and , . That splits into a drift , the part that moves along the gradient on average, and a jitter from noise left in the momentum.
Drift and jitter add as squares: RMS² = drift² + jitter².
A toy, at 1×: RMS 0.249, drift 0.100, floor √κ 0.229.
So the observed 0.2 to 0.4 doesn’t say how much of Adam’s update is noise. At the toy’s floor is 0.23: an RMS of 0.25 would be about 80% noise by squared size, and 0.4 about 30%. Which applies depends on , and we found no measurement of it on a frontier run. The factor describes one step. The noise in momentum is correlated across steps, so over many steps a coordinate’s random walk comes out about the same size whatever is (simulated: about 0.9 per square root of steps at 0.9, 0.95 and 0.975). A larger gives a smaller step, and not a smaller random walk.
Weight decay pulls weights toward zero each step. If you add it to the gradient, as L2 regularization, Adam’s division scales it too, and as Loshchilov and Hutter put it, weights with large gradients “do not get regularized as much”. Decoupled weight decay subtracts it outside the preconditioner, so every weight shrinks by the same fraction: . That is AdamW.
AdamW’s costs are the second buffer and a step that depends on noise. Its advantages are that it needs no communication beyond the gradient reduction a training system already does, and that it has no parameter it can’t handle.
Muon: a limit on how far a layer’s output moves
Take a layer and an update . Its output moves by , and that change is what the next layer sees. Bernstein’s derivation of Muon starts here. If an update moves a layer’s outputs too much, the network can destabilize, so pick the update that most reduces the linearized loss (the first-order change, , the sum of elementwise products) while bounding that output change:
The norm is the largest factor by which can scale the average entry size of an input, which equals the spectral norm (largest singular value) times . It bounds output change for inputs of RMS at most 1. Write the gradient’s singular value decomposition as . Then
and each is at least minus the spectral norm of . Setting every one to that limit gives
So the update keeps the gradient’s singular vectors and drops their sizes. is the orthogonal factor of the polar decomposition , the nearest matrix to the gradient with all singular values equal to 1. Since the objective is linear, the solution always sits on the boundary: the step has the full allowed size whatever the gradient. This is the normalized steepest descent direction, or in Frank-Wolfe terms the linear minimization oracle of the ball. “Trust region” is the common shorthand, with the same looseness as for Adam. Natural gradient and K-FAC measure output change on average over the data, using the input covariance. This bounds it for the worst input of a given size, and needs no statistics of the activations (our comparison).
Muon applies this to the momentum and not to the gradient, so the update is , computed on the short side of the matrix. Momentum doesn’t touch the bound, since any orthogonal update sits on the ball. It changes what’s being minimized: stands in for the gradient, lagged by about steps. It’s there for a reason. The polar factor is nonlinear, so the average of polar factors of noisy gradients isn’t the polar factor of the mean, and averaging first removes most of that bias. In the classical reviewer’s toy, with noise at twice the signal’s singular values, the signal direction keeps 64% of its size without momentum and 96% with it.
Two independent choices separate these optimizers: whether the geometry is per element or per matrix, and whether a running second moment moderates the step.
| No second moment | Second moment in the denominator | |
|---|---|---|
| Per element | Signum: | Adam: |
| Per matrix | Muon: | Shampoo, one-sided: , with an average of |
Shampoo is an older optimizer that accumulates such a matrix preconditioner. Jordan et al. note that Muon with momentum off is an “accumulation-free” Shampoo. Muon’s lower memory comes from having no denominator, which Signum shares, and it gives up the noise term for the same reason. A comparison of AdamW and Muon changes both axes at once. Wen et al. run relatives of all four cells on one model: after tuning, the sign-based Lion stays close to AdamW (under 1.2×), while Muon, Soap and Kron give about 1.3× under 520M. That points at the matrix axis and not at the denominator, on models this small.
The same frame gives the element-wise side a comparison with a caveat. Sign descent is steepest descent inside a max-norm ball, where each element may move by at most , and Adam reduces to it when its moving averages are switched off (Bernstein and Newhouse). With the averages on, Adam is a “cousin”. Those two limits allow very different updates. Take a update with element RMS , so that . With all of that in one singular value, as a sign update has when the gradient is rank one, it’s . Spread evenly, as in an orthogonal update, each is . So at equal element size the sign update can push the output times harder along one direction: 85 times at , V4-Pro’s width (computed), and on a rectangular matrix with short side . That’s a worst case. We found no measurement of how close real gradients come, though Jordan reports from manual inspection that SGD and Adam updates on transformer matrices “are almost low-rank matrices”.
The other side is first-order progress. Equal element RMS means equal Frobenius norm. Take a gradient that is rank one with equal-size entries, plus a small full-rank remainder. Its cosine with its sign update is near 1, and with its orthogonal factor near , because the factor lifts the remainder’s directions to full size and spends most of its length where the gradient is small. On that gradient, at equal RMS, the sign update reduces the linearized loss about times more. That needs a converged factor: with the remainder at a thousandth of the signal, five Newton-Schulz steps leave those directions small and the cosine is about 0.8 (computed, ). With no remainder at all, the factor is the sign update. Muon is the best step inside its own ball, and not per unit of weight movement. The case for it rests on what a loss does beyond first order and on noise, and neither the derivation nor these toys measure that.
Sign update (Signum): largest 32.00r, limit r·d = 32r; cosine with the momentum 0.97.
Orthogonal factor: largest 5.66r, limit r·√d = 5.66r; cosine with the momentum 0.37, or 0.33 after 5 Newton-Schulz steps with Jordan's coefficients.
Computing by SVD at every step would be too slow, so Muon uses Newton-Schulz iteration: divide the momentum by its Frobenius norm so all singular values lie below 1, then repeatedly apply an odd polynomial. Since , such a polynomial acts on each singular value and leaves and alone, using only matrix products. One training step is:
M ← βM + (1−β)∇ momentum, often in Nesterov form
X ← M / ‖M‖_F
repeat K times: X ← aX + b·XXᵀX + c·(XXᵀ)²X
W ← W − ηγX − ηλW γ is a scale, below; λ weight decay
Jordan et al. use and .
A flat spectrum after Frobenius normalization starts at 1/√s, 0.018 for s = 3072.
Labs use from 5 (GLM-4.5, Z.ai) to 16 (NVIDIA’s study), and Step 3.5, Qwen3.8 and NVIDIA use Polar Express, whose coefficients change per step by solving a minimax problem. The sources disagree on whether accuracy helps. Moonlight saw no gain from 10 steps over 5, and Amsel et al. found no gain beyond 6 or from an exact SVD on GPT-2 Small. Qwen found that 8 steps reduced gradient-norm spikes in a stress test, and NVIDIA reports slightly lower loss from exact polar factors on a 30B model. Number formats differ too. DeepSeek reports the iteration is stable in BF16 under its coefficients. StepFun saw rare unrecoverable spikes, simulations pointed at accumulated error in BF16 Polar Express at , and it moved the iteration to float16.
The step has nearly the same size however small the gradient is, so without weight decay nothing limits the weights’ growth. The provable bound is linear in steps. Pethick et al. observe growth as the square root of steps on CIFAR-10, for a close relative of Muon. Moonlight saw weights and layer outputs grow past BF16’s high-precision range when scaling up. With decay, the update is a Frank-Wolfe step: becomes a convex combination of its old value and , so decay sets a ball of radius that weights starting inside can’t leave (Pethick et al.), give or take the iteration’s overshoot. With RMS matching and that ball is far larger than real weights: element RMS at most 2, where Moonlight’s small-scale runs show under 0.1 early in training (computed). So there decay acts as a pull toward zero and isn’t a binding limit. Kolibri’s fixed decay gives a much smaller ball, about 4 in spectral norm at its peak rate, against about 2 at initialization (computed).
Whether it helps depends on the horizon. On an 800M model trained well past its optimal token budget, vanilla Muon led by 0.023 in validation loss at iteration 24,000, and decayed Muon by 0.017 at 66,000 (Moonlight). In Lion et al.’s 1B and 2.7B runs, Muon’s best result had no decay at all, though decay helped at 124M. Their Muown (used by Xiaomi from mid-training) treats each row’s magnitude as its own variable, a form of weight normalization, updated by Adam, and applies Muon to the direction. They report no spectral drift and lower perplexity than Muon in dense runs up to 2.7B. It needs a non-zero start for any matrix that would be zero-initialized, and its authors leave mixture-of-experts models open, which is where Xiaomi uses it.
What Muon costs to run
Jordan et al. count FLOPs per iteration on a matrix with long side and short side , at most . They count a forward and backward pass through the same layer at for tokens. So over iterations the overhead is at most , which, as Essential AI points out, shrinks with batch.
In a mixture-of-experts (MoE) layer, a router sends each token to of expert sub-networks, so an expert sees only about tokens. Take V4-Pro at . A dense matrix at its width of 7168 and the peak batch of 94.4M tokens gives 0.08%. An expert with hidden size 3072, active for 6 of 384 slots, sees 1.5M tokens and gives 2.1% (both computed). That’s the favourable end: on NVIDIA’s 72B model, with and an expert side of 2688, the same formula gives 7.3% at the 50M tokens its Muon ran at, and 14.7% at 25M (computed).
The bound also leaves out what a training system pays for: communication, waiting for the slowest rank, many small kernels, and number formats that differ between the forward pass and the iteration. Reported times come from other systems. Moonlight says usually 1% to 3% of forward-backward time at . Poolside reports less than 1% of the step on Laguna M.1, with communication overlapped with computation. DeepSeek prints no timing for V4-Pro.
Large training systems split each matrix across GPUs, and Muon’s update needs the whole matrix.
AdamW
Muon
Six labs describe a fix. DeepSeek assigns whole dense matrices to ranks with a knapsack packer, and reports under 10% memory overhead from padding on those when each rank manages at most five matrices. Moonshot’s K3 fetches only the shards each rank owns over peer-to-peer links. Z.ai restricts its all-gather the same way. Qwen’s Canzona balances estimated iteration FLOPs across ranks, and captures the many small kernels in a GPU command graph. Poolside gathers to one owner per parameter. StepFun reports about 5% less iteration time than its own naive all-reduce path, for under 4 GB of extra memory. Moonlight’s earlier design calculates communication at between 1 and 1.25 times distributed AdamW’s, which assumes the gather is in 16 bits.
The work brings new ways to fail. Poolside’s hash checks caught non-determinism in the iteration, “whose extra norm reductions and matmuls offer more opportunities for non-determinism than AdamW”, and it pinned the relevant kernels to deterministic algorithms.
What the measurements say
| Who ran it | Scale, tokens | Comparison | Muon vs AdamW |
|---|---|---|---|
| Pethick et al. (EPFL) | 124M; batch sweep with step size tuned per optimizer and batch | Validation loss | Scion “better at maintaining a low validation loss” than Muon and Adam as batch grows |
| Wen et al. (Stanford) | 130M to 520M swept per optimizer; 1.2B by a fitted law; dense | Tokens to a loss | 1.4× at 0.1B, 1.1× at 1.2B; Nesterov AdamW also about 1.1×; at 1.2B and 8× budget, mean of 10 benchmarks 66.98 vs 67.15 |
| Moonshot (Moonlight) | 399M to 1.5B, batch 0.8M to 2.1M tokens | FLOPs along a fitted law | “0.519x FLOPs”, the source of “2×” |
| Jordan et al. | 1.5B, short run, both tuned | Time to GPT-2 XL on HellaSwag | 10 vs 13.3 hours on eight H100s |
| Moonshot (Moonlight) | 2.24B active, 15.29B total; 1.2T of 5.7T tokens, before the learning rate decays | Same recipe | HumanEval 37.2 vs 29.3, MATH 19.8 vs 16.1, MMLU 60.4 vs 60.2, BBH 43.2 vs 45.3 |
| Pethick et al. | 3B GPT, far below a full budget, no seeds printed | Validation loss | 2.909 vs 3.024 |
| Essential AI | 100M to 4B; ratio plots to 10M tokens, a 500M sweep to 16M; dense | Tokens to a training loss | 10% to 15% fewer in its conclusion, which fits its plots at the lowest target losses; much more at high targets and large batches |
| NVIDIA | 3B active of 30B at 1×; 8B active of 72B at 2×; 1T to 3T tokens | Muon at 1× to 3× of 25M tokens vs AdamW at 1× | 72B: HumanEval 65.79 vs 57.62, Math 500 75.60 vs 80.20. 30B at 1×: Math 500 75.50 vs 71.50, HumanEval 62.56 vs 63.26 |
| Qwen | 28-layer 25B test model, Qwen3.5 structure, constant raised rate | Loss spikes per 10k steps | 0.2 vs 4.3 at twice the rate it calls optimal |
Higher benchmark scores are better. None of these comparisons runs as long as a shipped model: the longest is 3T tokens, against 15.5T for K2 and 33T for V4-Pro. MoE rows also give size two ways, and tokens per parameter differ by 7× to 10× depending on which you count, and by about 32× on V4-Pro (computed).
The 2× and the 1.1× are both FLOP ratios, since Essential AI notes that a token ratio is a FLOP ratio at fixed model size. They still measure different things. Wen et al. and Essential AI hold a model fixed and count tokens to a loss. Moonlight fits loss against compute as , with about 6 × parameters × tokens, along a path where model size and tokens grow together. Its batches are small, so its 2× is not a large-batch result.
A flat curve makes the ratio sensitive: for a gap , so each percentage point of gap moves the ratio by about 20% (computed). On this scale 1.1× is a gap of 0.5% and 2× is one of 4%, a difference of about 0.08 nats, with no interval printed on any fit. The baselines differ as well. Moonlight grid-searched AdamW’s learning rate and batch size and gave Muon the same values, so tuning Muon separately could only have helped Muon. What’s open is how strong the AdamW baseline was. Wen et al. sweep more settings per optimizer and write, of the literature generally, that “many reported speedups of 2× simply reflect a weak baseline”. Neither paper isolates how much of the gap this explains. Moonlight fits at compute-optimal budgets, 22 to 26 tokens per parameter by its Table 2, which Wen et al. call 1× Chinchilla (about 20), and Wen et al. state their 1.1× at 8×.
At 1.2B Wen et al.’s Nesterov AdamW reaches the same 1.1×, so in that range the speedup doesn’t need a matrix optimizer at all. They also predict from a fit that at 7B and 1× Chinchilla Muon would end with a higher loss than AdamW, by 0.09% on their printed coefficients (computed), six times past their largest model. That’s effectively a tie. Above 1.2B the pre-training comparisons we found have Muon ahead: Jordan et al. on time to a benchmark (both tuned, short run), and Pethick et al., Essential AI, Moonlight and NVIDIA on loss. Their benchmarks are less uniform. Moonlight and NVIDIA didn’t tune each optimizer separately and Essential AI doesn’t say. No study we read has both per-optimizer tuning and that scale.
Large batches are where Muon’s case is strongest. Essential AI sweeps batch from 128K to 16M tokens on a 500M model, and finds Muon “more effective than AdamW in retaining data efficiency at large batch sizes”. NVIDIA reports AdamW not benefiting from 50M-token batches on its model, without re-tuning AdamW’s betas or . Its Muon at 2× runs against AdamW at 1× over the same tokens, so Muon gets half the updates and its loss result is the stronger for it. AdamW at 2× has its own chart, without Muon on it. NVIDIA hypothesizes that with less noise AdamW’s second-moment estimate becomes “poorly calibrated”. We found no study that isolates this. In the toy above a re-tuned learning rate would cancel Adam’s rising RMS, so that explanation can only separate runs that keep one rate. The one protocol that tunes the step size for each batch and optimizer is Pethick et al.’s, at 124M.
Sources differ on whether Muon needs a batch ramp. On a 10.8B test model, Qwen’s run ramping to 25.2M tokens ended within run-to-run variance of starting there and cost 18.8% more steps, so it doesn’t ramp in production. No AdamW ramp was run. NVIDIA ramps both optimizers for its larger batches and credits the ramp for Muon’s lead. DeepSeek V4, Z.ai and StepFun schedule their batches too, while DeepSeek V4.1, like Qwen, runs its batch fixed from the start.
NVIDIA’s MoE argument cuts in an unexpected direction. Experts “remain in a lower-effective-batch regime”, and raising the global batch “primarily stresses” the dense parts. On V4-Pro an expert sees 1.5M tokens against 94.4M for dense parts. Experts also hold most of the parameters and have the worst overhead bound. So the large-batch evidence bears mainly on a small share of the matrices (our reading). We found no study that isolates this, for instance by running Muon on dense parts alone.
KL-SOAP goes further along the matrix axis: a Shampoo-style method with Kronecker factors and an eigenbasis, at a “much larger” memory cost. NVIDIA finds it has a “slight edge” over Muon and recommends it where memory isn’t limiting, after fixing a stale-basis instability. Wen et al. find Muon “outperformed by Kron and Soap” at 8× Chinchilla or more.
On stability, DeepSeek V4 names “improved training stability” with no measurement, and the same report describes loss spikes tied to MoE outliers and fixes them by other means. On the 25B test model at 4× the rate it calls optimal, Qwen’s AdamW spiked 183 times per 10k steps and hit the clipping threshold on 213 of 19,932 steps, while Muon never hit it (its spike rate there isn’t printed). The same report toggles a gating layer under AdamW at another rate and takes spikes from 32.0 to 3.2 per 10k. So architecture moves that number by the same order, 10× against about 21× for the optimizer at 2× (computed), and those figures aren’t on one ladder, since the report’s figures imply different optimal rates.
Moonshot’s K2 report says exploding attention logits occur “more frequently with Muon but less with AdamW”. On a 53B test model with plain Muon they “quickly exceed a magnitude of 1000”, and K2 was trained with an added clip on queries and keys, which it calls MuonClip. GLM-5 and Qwen3.8 report no such clip, the first crediting its head split. Moonlight’s appendix says AdamW “demonstrated healthier behavior” on that metric. AdamW runs diverge too: NVIDIA traces one Nemotron 3 Ultra divergence to a precision change in the output layer, and calls a second undetermined. None of the reports we read describes abandoning Muon or calls an AdamW run a mistake, and published recipes describe what shipped.
Parameters that aren’t linear maps
Bernstein’s derivation assumes dense inputs and a matrix mapping activations to activations, and he says that “separates the Linear layer from, say, an Embedding layer that takes one-hot inputs”. Several kinds of parameter fall outside it, or stay off Muon anyway.
- Scalars and vectors have nothing to orthogonalize: norm scales, biases, the per-head decay and beta projections in Qwen3.8’s Gated DeltaNet layers.
- An embedding table takes one token at a time. The inputs that occur are one-hot vectors, and on them the output change is one token’s vector. The spectral norm also limits outputs on mixtures of tokens, which never occur, so it constrains the wrong set (our reading of Bernstein’s exclusion).
- The LM head has a dense input, so the theory doesn’t exclude it. Jordan still keeps it on AdamW, and says this “does not seem to follow from the theory, and is instead driven by empirics”.
- The router’s outputs are scores for separate experts. Qwen saw Muon destabilize it early in training, with no gain later, and offers the independence of those scores as “one possible explanation”.
- Qwen’s low-rank projections do better on AdamW, which it attributes to “their very elongated shape”.
- NVIDIA saw Mamba’s Conv1D filters give about 0.1% higher loss under Muon, and “sometimes” go to NaN. Its reason is that they’re shared and re-applied at every position.
| Group | Kolibri | DeepSeek V4 | DeepSeek V4.1 | GLM-4.5 | Qwen3.8 |
|---|---|---|---|---|---|
| Attention Q/K/V, experts | Muon | Muon, by exclusion | Muon | Muon, experts by exclusion | Muon |
| Token embedding | Adam | AdamW | Sinkhorn-momentum | not Muon, unnamed | AdamW |
| LM head | AdamW | AdamW | Sinkhorn-momentum | Muon, by exclusion | AdamW |
| MoE router | AdamW | Muon, by exclusion | Muon, by exclusion | AdamW | |
| Norm scales | Adam | AdamW | AdamW | not Muon, unnamed | |
| Gates, scaling factors | AdamW (mHC) | AdamW | AdamW (output gates) | ||
| N-gram table | Sinkhorn-momentum | Adam, no decay |
“By exclusion” means the report lists what isn’t on Muon and says everything else is. Kolibri’s “Adam” groups carry no weight decay, which makes them the same update as AdamW. MiMo-V2.6 keeps embeddings, head and router on AdamW from mid-training. So labs split the parameters differently, and they put the router and head on different optimizers.
Labs that keep AdamW run the two optimizers together in one of two ways, and a third approach removes it from most of these groups.
The first matches update sizes. Moonlight’s Lemma 1: a full-rank orthogonal update on an matrix has element RMS , so large matrices get small updates. Moonlight multiplies by to fix RMS at 0.2, in AdamW’s observed range, so that one learning rate and one decay can serve both. DeepSeek uses 0.18. Poolside, in preliminary experiments, ran Muon without this rescaling and with standard decay coefficients, and its experts collapsed from about 450B tokens, spreading layer by layer through the first three MoE layers until divergence. That run changed two things together and had no AdamW arm. Poolside’s explanation is that rescaling lets Muon run “at AdamW-scale LRs”, removing “the order-of-magnitude mismatch in effective WD”. The cost is that RMS matching gives up the output-change limit that motivates Muon, and it ties 0.2 to AdamW’s betas, as the Adam figure shows. Bernstein writes of his own factor, “I’d actually expect the one here to scale better”.
The second gives each group its own rate and makes Adam scale-free. Kolibri (Aleph Alpha) uses Bernstein’s factor and separate rates per group: 1e-3 for Muon, 2.5e-5 for router and head, 2e-3 for embeddings and norms. It sets = 1e-15 so that Adam’s updates on small-gradient parameters “stay independent of the gradient scale, as Muon updates are by construction”. The cost is three rates and a per-step decay to tune and carry through every stage.
The third drops Adam and normalizes these parameters directly. If the argument for Muon is to choose a norm that suits the layer, an embedding has its own best norm. For one-hot input the natural limit is on each token’s vector separately, and normalizing each vector is exactly the best step inside that limit. This is a worst case over tokens: it treats a token seen once like one seen a million times, and its direction then rests on few samples. Adam already gives a rare token a similar size on the steps it appears, though its average per step falls with the root of its frequency. What normalization gains is direction: on a row with Gaussian entries it beats the sign update by about 1.25× in linear decrease at the same element RMS, and by more for heavier tails (our computation). An orthogonal factor doesn’t equalize tokens at all: row of has squared norm , its leverage score, which follows frequency.
Rows: tokens, most frequent first. Bars: each token's update size.
Scion (Pethick et al.) works out these choices per layer as conditional gradient, the family of Frank-Wolfe. For one-hot input it recommends normalizing each token’s vector, for the last layer the sign, and with tied weights the sign on both ends. Its reported GPT results use the tied form, so they test a norm-based optimizer without Adam, and not per-token normalization. Orvieto and Gower note that Scion’s architecture has no trainable gain in its normalization layers, and that “experiments in the usual setup are needed”. Its memory claim, one set of gradients in half precision and no separate momentum buffer, should be weighed against production runs that keep gradient accumulation in FP32 for stability.
DeepSeek V4.1 is the production case, and it goes past this frame. It applies Nesterov momentum and then Sinkhorn balancing to embeddings, head and Engram tables (its name for n-gram tables): alternately normalize rows (one token) and columns (one feature), 11 passes, then scale by 0.18 to match Adam’s size. The column passes mean the update is no longer the best step inside any fixed ball. It is a diagonal scaling of the momentum on both sides, so read it as a preconditioner, a diagonal relative of Shampoo. V4.1 uses no weight decay on these groups, and says this “requires only a momentum buffer while empirically outperforming Adam”. It prints no number, and it’s the only lab we found stating this. SinkGD (Scetbon et al.), which V4.1 cites, is a different setup: stateless, on linear layers, with Adam kept on embeddings, norms and head. Norm scales and biases stay on AdamW at V4.1, so no shipped model we read states that it runs without Adam.
After pre-training
A recipe continues after pre-training with mid-training, then supervised fine-tuning (SFT) on prompt-and-response examples, and often reinforcement learning (RL) against a reward (01). Jordan’s post asked whether Muon “works only for pretraining”. Moonlight gave the first answer, fine-tuning checkpoints taken partway through pre-training, at 1.2T of 5.7T tokens, for two epochs on an open SFT mixture:
| Pre-train → SFT | MMLU | HumanEval | MBPP | GSM8K |
|---|---|---|---|---|
| Muon → Muon | 55.7 | 57.3 | 55.6 | 68.0 |
| AdamW → Muon | 55.3 | 53.7 | 55.5 | 62.1 |
| Muon → AdamW | 50.2 | 52.4 | 55.2 | 64.9 |
| AdamW → AdamW | 52.0 | 53.1 | 55.2 | 64.6 |
Muon through both stages is best on every column, and no row has seeds or error bars. On a Qwen2.5-7B base, Moonlight’s Adam-SFT beats Muon-SFT on all four benchmarks, GSM8K 89.8 to 85.8, a result it calls “on par”. It concludes that Muon is better applied in pre-training than in SFT, and names an “optimizer mismatch” as an open problem: the best SFT optimizer depending on which optimizer pre-trained the checkpoint. Later studies disagree about how large that effect is, and they use different metrics.
SFT with AdamWSFT with Muon
Moonlight GSM8K
Moonlight MMLU
Qu et al. NanoChat
Zhao et al. 1.2B at 1× Chinchilla
Zhao et al. 1.2B at 8× Chinchilla
Qu et al. (ICML 2026) find a symmetric mismatch on a 561M model pre-trained from scratch, then test why. They link it to implicit bias, the kind of solution each optimizer tends toward: for sign descent on a linear regression toy the minimum max-norm solution, and for Muon the minimum spectral-norm one. They offer this as a hypothesis, with theorems about the toy, started from zero, with no momentum and step sizes going to zero. Sign descent stands in for Adam, and no theorem covers a change of optimizer partway. They then show that the damage grows with update strength. Under LoRA (low-rank adapters that limit how much weights can change), the perplexity gap on that 561M model falls by 39% and 78%, and pooled across their T5, Llama and CLIP tasks by 0.72%. LoRA-Muon matches LoRA-Adam on Llama 2-7B, though full fine-tuning with Adam stays above both, a mean score of 54.92 against 52.15.
Zhao et al. (a NeurIPS 2026 workshop paper) find “little evidence of a meaningful optimizer mismatch penalty”. They select the learning rate per setting on held-out loss and report three trials. Final scores are “largely explained by the independent effects” of the two optimizers, with Muon SFT ahead on AdamW-pretrained open models in 4 of 5 settings up to Qwen3-14B, mostly by less than run-to-run variation. Their result on Qwen2.5-7B disagrees with Moonlight’s on the same base and the same dataset name. The two differ in tuning and in evaluation (GSM8K scores sit about 10 points apart), so we can’t say which explains it.
Li et al. use a different measure, a trade-off between learning the new task and keeping the old, on models up to 600M, and find that AdamW overtakes Muon after fine-tuning once the pre-training budget is large enough, and earlier for larger models. Bagai et al. report, in a workshop abstract (we couldn’t read the paper), that on Qwen3-4B Muon trails Adam until the learning rate of its auxiliary AdamW group is tuned, after which the two are “indistinguishable”.
These sources don’t settle it, and none tests a mixture-of-experts checkpoint at frontier scale. Across them, both of Muon’s rates need tuning at SFT, its own and that of the parameters still on Adam, and a comparison at one fixed schedule is unreliable.
Of the seven labs naming an optimizer both in pre-training and at a later stage, five keep one family: Aleph Alpha, Moonshot (K2), Poolside and StepFun on Muon in SFT, NVIDIA on Adam. Two switch. Mistral states Muon for RL, with no weight decay, and doesn’t state its mid-training or SFT optimizer. Xiaomi moves to Muown in mid-training “to prepare the model for subsequent large-batch RL”, because its experiments “showed diminishing optimization efficiency as the batch size increased in mixed-task RL”, with no number printed. It reports “no loss spike throughout the training process”. That statement covers spikes, and not the end quality the fine-tuning studies measure. Ruan et al., on Qwen2.5 models up to 3B on ALFWorld with five seeds per cell, tie much of Muon’s late gain in RL to a larger effective step: an RMS-matched control removes it, and tuned AdamW nearly matches at 3B on one objective. AdamW couldn’t take that step directly, since higher rates failed under the shared recipe, though they note joint tuning with regularization might change that. Fan et al. report the opposite outcome on single-turn RL: on Qwen3-1.7B and 4B, at one learning rate shared across optimizers over 40 to 80 steps, Muon’s accuracy falls to near zero on MATH and GSM8K while AdamW improves. They blame whitening, which lifts noise directions in low-signal gradients to full size, and propose a filtered replacement. They print no seeds or rate sweep, and Ruan et al. read that failure as rate dependence. Six labs state Muon or a variant in RL and report no collapse. So the labs’ RL runs and these two studies don’t yet fit one explanation.
Reading a recipe
Learning rates compare only under one convention. RMS-matched multiplies by , with about 0.2: Moonlight, K2, DeepSeek (0.18), Laguna, Qwen3.8, GLM-4.5. Bernstein’s multiplies by : Kolibri, and probably Mistral in RL, which says “spectral scaling” with no formula. Jordan’s original form, , survives in smaller studies. On a square matrix RMS matching and Bernstein’s differ by at the same rate, about 17 at (computed).
| × learning rate | element RMS | largest singular value | output-change limit |
|---|---|---|---|
| raw orthogonal | 1.18e-5 | 1.00e-3 | 1.00e-3 |
| RMS-matched 0.2·√ℓ | 2.00e-4 | 1.69e-2 | 1.69e-2 |
| RMS-matched 0.18·√ℓ | 1.80e-4 | 1.52e-2 | 1.52e-2 |
| Bernstein √(d_out/d_in) | 1.18e-5 | 1.00e-3 | 1.00e-3 |
Weight decay compares only under a stated formula. Where a formula is printed (K2, DeepSeek V4, and the Moonlight and NVIDIA papers), decay is per step, and six of the 11 labs state 0.1. As frameworks implement it, AdamW’s decay also scales with the learning rate, and Essential AI calls that “coupled”, a different sense from “decoupled” above. Kolibri decays by per step, independent of the learning rate. Inkling (Thinking Machines) “coupled the weight decay strength to the square of the learning rate”, pointing to Kosson et al. and Defazio without giving a formula. In RL, Aleph Alpha, Mistral and Xiaomi use none, and StepFun keeps 0.1.
Many Muon setups treat Q, K and V as separate matrices, as Jordan advised, and several split further by head: GLM-5’s Muon Split, K3’s Per-Head Muon, V4.1’s head-wise Muon, and Qwen3.8’s split fused parameters. Qwen’s argument is that one orthogonalization over a concatenated matrix mixes singular directions across unrelated sub-blocks. Splitting limits each head’s output and gives up the limit on the layer: the concatenated update’s largest singular value can rise by up to at the same element RMS, for heads (classical reviewer’s computation). K3 says the iteration gets slightly cheaper, and Qwen says the step fragments into many small kernels. Only GLM-5 prints numbers for a head split: on its table, one attention variant with Muon Split beats its comparison on four benchmarks and trails on three, which it calls matching. NVIDIA tested the earlier step, splitting a fused Q, K and V tensor, on a 30B model. Loss was lower early, and “the losses at the end of the full 1T-token horizon are identical”. That is loss alone, and it doesn’t test splitting by head.
The options side by side
| Optimizer | State per parameter | Step cost | Sharding | Evidence | Labs in pre-training |
|---|---|---|---|---|---|
| Signum | 1 buffer | element-wise | anywhere | about a third more budget than Adam (160M) | |
| AdamW | 2 buffers | element-wise | anywhere | the baseline; a noise-scaled step | 3 of 11 |
| Muon | 1 buffer | iterations per matrix | whole matrix | see the table above | 8 of 11 |
| Muown | 1 buffer plus four short vectors per layer | about 1.5% over Muon, nothing when sharded | as Muon | lower perplexity than Muon to 2.7B (its authors) | Xiaomi, from mid-training |
| Momentum + Sinkhorn | 1 buffer | row and column norms, passes | not stated | “outperforming Adam”, no number | DeepSeek V4.1, on embeddings and head |
| Scion | no buffer beyond the gradient | as Muon, sign elsewhere | as Muon | 3B without Adam, no trainable norm gain | |
| KL-SOAP | “much larger” | matrix products, eigenbasis updates | matrix-based | “slight edge” over Muon on a 30B model |
Where the models in the map sit
| Lab | Latest model stating it | Pre-training | Later stages, where stated |
|---|---|---|---|
| Aleph Alpha | Kolibri-1 | Muon + Adam + AdamW | same groups in SFT; RL moves head and router to Adam |
| Moonshot | K3 | per-head Muon | K2: Muon in SFT; K2.5: MuonClip in RL |
| DeepSeek | V4.1 (V3 was AdamW) | Muon + AdamW + Sinkhorn-momentum | |
| Z.ai | GLM-5 | Muon | |
| Poolside | Laguna | Muon + AdamW | Muon “across all training stages” |
| Qwen | 3.8 (3.5’s AdamW is implied by the 3.8 report) | Muon + AdamW + Adam | |
| Thinking Machines | Inkling | Muon + Adam | |
| StepFun | Step 3.5 Flash | Muon | Muon in SFT and RL |
| Mistral | Large 4 | AdamW | Muon in RL; others not stated |
| Xiaomi | MiMo-V2.6 | AdamW | Muown on hidden matrices from mid-training |
| NVIDIA | Nemotron 3 Nano, Super (Ultra inherits) | AdamW | Adam in 3.5 SFT, AdamW in quantization-aware distillation |
So 8 of the 11 labs that state a pre-training optimizer use Muon. StepFun’s and NVIDIA’s statements are in reports outside our corpus; on corpus documents alone the count is 7 of 9. Liquid uses AdamW in SFT and doesn’t state its pre-training optimizer. OpenAI, Google, MiniMax and Tencent don’t state one in the documents we read. NVIDIA’s Muon paper is a research study and doesn’t establish what any shipped Nemotron used.
Choosing
This is our judgment, and the sources don’t settle it. For a new large model trained from scratch at large batches, on a system you can change, we would start with Muon on linear maps. In the comparisons above 1.2B it comes out ahead on pre-training loss or time, its large-batch evidence comes from two groups’ sweeps (Essential AI, NVIDIA), and most labs chose it, though few print a comparison of their own. Against it, AdamW leads on math at NVIDIA’s 72B and on BBH at Moonlight’s 2.24B active, Wen et al. find 1.1× and no downstream gain at 1.2B with Nesterov AdamW level, and Li et al. find the ordering reverses after fine-tuning at up to 600M.
Don’t build a budget on 2×. It comes from one fitted law, at batches around 2M tokens. Don’t build one on 1.1× either: that stops at 1.2B and dense models. No study measures the gain at your scale, so if the engineering is what’s being decided, run your own comparison at your batch, with each optimizer tuned. Keep AdamW or Adam for scalars and norms, and test the head, router and embeddings on your own architecture. If you have no reason to tune each group, RMS matching is the convention with the most users.
AdamW remains defensible at small batches, on a training system you can’t change, or wherever the budget doesn’t pay for the engineering. Three labs ship it in pre-training, and Wen et al. report that a carefully tuned baseline leaves 1.1× at 1.2B and that Nesterov AdamW reaches about the same. Nesterov and equal betas are the cheap things to try first. Raising to reach equal betas shrinks the step, so the learning rate needs re-tuning.
Moving a checkpoint to Muon later has one stated reason, large-batch RL at Xiaomi, with no number printed. Ruan et al. tie much of the measured gain to a larger step that AdamW couldn’t sustain, and Fan et al. show Muon collapsing outright at a rate shared with AdamW, so screen the rate before committing. Plain SFT is an open question. Tune every group’s learning rate for whichever optimizer you pick, and treat a comparison at one fixed schedule as unreliable.
Easy to get wrong
- “Muon is 2× better.” 2× is FLOPs along one fitted law, where model size and tokens grow together, at batches around 2M tokens. Holding the model fixed, Essential AI’s summary has 10% to 15% fewer tokens, matching its plots at the lowest target losses, and Wen et al. have 1.1× at 1.2B.
- “NVIDIA shows AdamW failing at Muon’s batch.” Its Muon at 2× runs against AdamW at 1× over the same tokens. That makes the loss result stronger. AdamW at 2× did run, on a separate chart with no gain, but its betas and weren’t re-tuned, and AdamW still leads on math at 72B.
- “Muon halves training memory.” It halves optimizer state, which on FP32 weights and gradients is about a quarter of the per-parameter total. Gathering whole matrices raises the peak on a rank.
- “8 of 11 labs tested it and chose Muon.” Two of the eight print a comparison of their own. The other 6 cite others, state an observation or give no reason, and 3 labs ship AdamW.
- “Muon fixed stability.” Qwen’s gate has an effect of the same order under AdamW, and K2 needed a clip because of Muon, on attention logits where it and Moonlight report AdamW better.
- “A Muon learning rate of 1e-3 is higher than one of 2e-4.” On a square 7168 matrix, Kolibri’s convention gives an output-change limit of 1e-3, and DeepSeek’s gives about 3e-3 (computed).
Check yourself
A gradient on a 1024 × 1024 matrix is rank one with equal-size entries, plus a tiny full-rank remainder. At equal element RMS, how do its sign update and its exact orthogonal factor compare on largest singular value and on first-order loss decrease?
The sign update has largest singular value and the orthogonal factor , so the sign update can move an output 32 times more. Its cosine with the gradient is 1 against , so it also reduces the linearized loss about 32 times more. Muon gives up first-order progress to bound output change. With no remainder at all, the two are the same matrix.
A 400B model trains at FP32. How much optimizer state does it hold under AdamW and under Muon, ignoring the groups Muon doesn’t cover?
3.2 TB and 1.6 TB: 400B × 4 bytes × 2, and × 1. Muon’s parameters that stay on Adam bring some of that back.
With 5 Newton-Schulz iterations, a 10M-token batch and 8 of 256 experts active, what does Jordan’s bound give for a dense matrix with short side 2048 and for an expert with short side 2048? Why do they differ?
0.10% and 3.3%: , and the same over = 312,500 tokens. An expert costs as much to orthogonalize but sees far fewer tokens, so its forward and backward pass is smaller next to it.
Why doesn’t a table of Muon → Muon beating every other row show that a mismatch exists?
It could also mean Muon is better in each stage independently. A mismatch means the best SFT optimizer depends on the pre-training optimizer, so you need to compare the SFT effect on each checkpoint. On Moonlight’s GSM8K the effect of Muon SFT flips sign (+3.1 and −2.5), and on its MMLU it doesn’t (+5.5 and +3.3).
In the map
Methods
- Muon 13 sources
- AdamW 7 sources
- Nesterov Muon 1 source
- Spectral scaling convention for Muon 1 source
- Moonlight-style learning-rate scaling 1 source
- Hybrid Newton–Schulz iterations 1 source
- Newton–Schulz orthogonalization 1 source
- Muon Split 1 source
- Per-Head Muon 3 sources
- Splitting fused parameters before orthogonalization 2 sources
- Category-specific optimizer assignment 5 sources
- Decoupled weight decay 2 sources
Taxonomy nodes
- optimizer 29 methods
Further reading
- Jordan et al., Muon: the algorithm, the FLOP count, and the coefficient search.
- Bernstein, Deriving Muon: the output-change derivation in full. Bernstein and Newhouse, Old Optimizer, New Norm: Adam and Shampoo as steepest descent with averages off.
- Orvieto and Gower, In Search of Adam’s Secret Sauce: over 1,500 models, equal betas, and the mean and variance reading.
- Loshchilov and Hutter, Decoupled Weight Decay Regularization: the original AdamW argument.
- Liu et al., Muon is Scalable for LLM Training: Moonlight’s scaling, distributed design and SFT tables.
- Essential AI, Practical Efficiency of Muon for Pretraining: compute against time as batch grows.
- Wen et al., Fantastic Pretraining Optimizers: a tuned benchmark with the smallest gains.
- Khona et al., SOAP, Muon, and Beyond: large MoE comparisons and KL-SOAP.
- Lion et al., Muown: the spectral-norm decomposition behind row control.
- Pethick et al., Scion: norm choices per layer as conditional gradient. Scetbon et al., SinkGD: Sinkhorn on linear layers.
- Amsel et al., The Polar Express: minimax coefficients.
- Ruan et al., When Does Muon Help Agentic Reinforcement Learning? and Fan et al., Rethinking Muon Beyond Pretraining: RL, with opposite outcomes.
- Qu et al., Can Muon Fine-tune Adam-Pretrained Models?; Zhao et al., The Ephemeral Cost of Switching Optimizers; Li et al., Pretraining Data Scale Reverses the Muon–AdamW Fine-Tuning Gap: read together, for the disagreement.
