NVIDIA
NVIDIA-Nemotron-3.5-Lightning-30B-A3B
What NVIDIA-Nemotron-3.5-Lightning-30B-A3B’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 30B / 3B active
- Context
- 1M
- Attention
- Mamba hybridinterleaved Mamba-2 and MoE layers with select GQA attention layers
- Experts
- 128 routed + 1 shared, top-6
- Input
- text
- Licence
- OpenMDW-1.1
- Released
- 2026-08-11
Against the consensus recipe
Its documents state 10 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationused
- Synthetic data generationnot in its documents
- Corpus deduplicationused
Training objective
- Multi-Token Predictionused
- Knowledge insulationnot in its documents
- Quantization-aware distillationused
Model architecture
- Mixture of expertscore
- Multi-Token Predictioncore
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
Inference & serving
- Thinking modenot in its documents
- Multi-Token Predictionused
- FP8not in its documents
Checked in its code
Its code shows 6 of the 19 architecture features checked — 4 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_nemotron_h.py, configuration_nemotron_h.py) and config.json at revision a9904d24bc, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention code only | in its code | not stated | Full-attention layers project 32 query heads but only 2 key/value heads, which are repeated via repeat_kv for the group-wise attention… show evidenceFull-attention layers project 32 query heads but only 2 key/value heads, which are repeated via repeat_kv for the group-wise attention computation. num_attention_heads = 32num_key_value_heads = 2 |
| Multi-head latent attention | not in its code | not stated | There is no KV down-projection to a shared latent with per-head up-projection anywhere in the attention code; NemotronHAttention uses… show evidenceThere is no KV down-projection to a shared latent with per-head up-projection anywhere in the attention code; NemotronHAttention uses standard separate q/k/v projections. |
| Sliding-window attention | not in its code | not stated | sliding_window is null in this config, so no attention layer restricts tokens to a local window. show evidencesliding_window = null |
| Interleaved sliding-window and global attention | not in its code | not stated | layers_block_type only distinguishes mamba/moe/attention (full_attention), with no sliding-window layer type or pattern present. show evidencesliding_window = null |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | not in its code | not stated | No indexer-scoring mechanism or top-k token selection code exists in the given files, and no index_topk/index_n_heads config keys are… show the whole verdictNo indexer-scoring mechanism or top-k token selection code exists in the given files, and no index_topk/index_n_heads config keys are present. |
| Linear-attention or state-space layers alongside full attention code only | in its code | not stated | layers_block_type interleaves mamba (linear_attention) layers with full_attention and moe layers across the 52-layer stack. show evidencelayers_block_type = ["mamba", "moe", "mamba", "moe", "mamba", "attention", "moe", "mamba", "moe",… |
| Gated DeltaNet layers | not in its code | not stated | The recurrent layers implement the Mamba-2 selective SSM scan, not a gated delta rule; no delta-rule write or decay gate of that form… show the whole verdictThe recurrent layers implement the Mamba-2 selective SSM scan, not a gated delta rule; no delta-rule write or decay gate of that form exists. |
| Mamba-2 layers | in its code | adopted | The 'mamba' layers use NemotronHMamba2Mixer implementing the Mamba-2 SSD chunked scan with per-head scalar decay (A_log, D, dt_bias) and… show evidenceThe 'mamba' layers use NemotronHMamba2Mixer implementing the Mamba-2 SSD chunked scan with per-head scalar decay (A_log, D, dt_bias) and ssm_state_size. mamba_num_heads = 64ssm_state_size = 128 |
| Learnable attention sink | not in its code | not stated | The eager/sdpa attention implementation has no extra learned sink logit column added to the softmax denominator. show evidence |
| Gated attention output | not in its code | not stated | The attention output (NemotronHAttention) is passed directly to o_proj with no sigmoid gate multiplication; gating only appears in the… show evidenceThe attention output (NemotronHAttention) is passed directly to o_proj with no sigmoid gate multiplication; gating only appears in the Mamba mixer's silu gate, not in softmax attention. |
| QK normalization | not in its code | not stated | NemotronHAttention projects q/k/v and applies no RMSNorm/LayerNorm to queries or keys before the dot product. show evidence |
| Partial RoPE | code unclear | not stated | partial_rotary_factor is 1.0 (full rotary) but the attention forward code does not actually call apply_rotary_pos_emb or build rotary… show evidencepartial_rotary_factor is 1.0 (full rotary) but the attention forward code does not actually call apply_rotary_pos_emb or build rotary embeddings in the given excerpt, so it's unclear whether/how rotary is applied at all for this model's attention layers. partial_rotary_factor = 1.0 |
| YaRN RoPE scaling | not in its code | not stated | No rope_scaling key is present in this config, so no YaRN scaling is configured. |
| Layers without positional encoding (NoPE) | code unclear | adopted | The attention module's forward does not show rotary embedding application at all (no cos/sin passed into NemotronHAttention.forward), so… show evidenceThe attention module's forward does not show rotary embedding application at all (no cos/sin passed into NemotronHAttention.forward), so it cannot be determined whether some full-attention layers skip positional encoding while others apply it. |
| Mixture of experts | in its code | adopted | NemotronHMoE routes tokens via NemotronHTopkRouter to a subset (top-6 of 128) of NemotronHExperts. show evidencen_routed_experts = 128num_experts_per_tok = 6 |
| Shared expert code only | in its code | not stated | Each MoE layer includes a NemotronHMLP shared expert of intermediate size 3712, applied to every token in addition to routed experts, with… show evidenceEach MoE layer includes a NemotronHMLP shared expert of intermediate size 3712, applied to every token in addition to routed experts, with n_shared_experts=1. n_shared_experts = 1moe_shared_expert_intermediate_size = 3712 |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | The router adds e_score_correction_bias to the sigmoid scores used only for top-k expert selection, while the actual weights gathered for… show evidenceThe router adds e_score_correction_bias to the sigmoid scores used only for top-k expert selection, while the actual weights gathered for output weighting come from the unbiased scores. |
| Multi-token prediction layers documents disagree | not in its code | adopted | num_nextn_predict_layers is 1 in config but is not actually wired to build MTP modules in the given CausalLM/Model code (no mtp module… show evidencenum_nextn_predict_layers is 1 in config but is not actually wired to build MTP modules in the given CausalLM/Model code (no mtp module construction appears, and loading explicitly ignores mtp.* keys), making the modules effectively absent from this build. num_nextn_predict_layers = 1 |
| SwiGLU feed-forward | not in its code | not stated | Both the dense NemotronHMLP and the MoE expert MLP use a single activated up_proj followed by down_proj with relu2 activation (squared… show evidenceBoth the dense NemotronHMLP and the MoE expert MLP use a single activated up_proj followed by down_proj with relu2 activation (squared ReLU), not a SiLU-gated two-branch SwiGLU. mlp_hidden_act = "relu2" |
token mixer 5
“each of the five layers of the drafter body wraps the attention and MLP sublayers in a grouped dynamic convolution”
softmax attention 1
sliding window attention 1
“We therefore serve it with sliding-window attention over the most recent 8,192 positions in every drafter layer”
linear attention & state space 1
Mamba 1
“NVIDIA uses Mamba-2 for much of the sequence processing”
hybrid layer stacking 2
“both models combine large-scale distillation, long-context training, and native speculative decoding”
“The model employs a hybrid Mixture-of-Experts architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention…”
channel mixer 5
mixture of experts 5
“NVIDIA Nemotron 3.5 Lightning is an open 30B mixture-of-experts (MoE) model with 3B active parameters built for that execution layer of…”
“MoEs are fast and efficient because a router sends each token to just a few of its many experts, so only a fraction of the model’s…”
expert load balancing 1
“Router-bias updates are explicitly disabled.”
latent mixture of experts 2
“routed to an expanded set of experts that operate entirely in this latent space, and then projected back”
“we shrink the routed expert input dimension to reduce communication and memory costs, and reinvest the saved capacity into increasing the…”
positional encoding 1
“Nemotron 3 models do not use RoPE in attention layers and therefore do not suffer from out-of-distribution RoPE issues during context…”
prediction head 7
“The model includes Multi-Token Prediction (MTP) layers, which predict multiple future tokens to provide richer training signals.”
“the backbone remains frozen and only the MTP head parameters are updated.”
“adopting the recursively shared MTP formulation”
“input hidden states are sampled from the sequence of states produced by earlier MTP steps 1,...,k − 1, rather than relying exclusively on…”
“We first fuse the latent MoE projections into the experts’ up- and down-projection matrices, then prune the hidden size from 4096 to 2688”
“conduct a dedicated MTP warmup phase for 100 B tokens, freezing the backbone parameters and optimizing only the MTP head”
“We apply a dedicated MTP-only training recipe at two points”
filed at the root 1
“We replace it with a calibration term.”
multi-token prediction objective 4
“Post-training MTP boosting improves average speculative-decoding acceptance length”
“predicting multiple future tokens provides richer training signals and encourages models to plan several steps ahead”
“NVIDIA added a dedicated continued-pretraining stage for these MTP layers.”
“Whereas Nemotron 3 Ultra uses a temperature-scaled forward-KL objective, we now use the LK loss”
distillation objective 8
“The student minimizes the reverse KL𝐷KL(𝜋𝑆(·|𝑐𝑡)‖𝜋𝑇 (·|𝑐𝑡))at every position of its own responses”
“The released checkpoint uses the predefined aggressive PTQ recipe followed by SFT-based QAD at 200 iterations.”
“with Ultra 3 Base as the teacher and Skew Reverse KL divergence as the distillation loss function”
“these runs use a knowledge-distillation loss with the frozen backbone as the teacher.”
“we combined the KL loss with cross-entropy loss, assigning them weights of 0.9 and 0.1, respectively”
“We used standard forward KL divergence loss along with an extra sink token to capture the residual probability mass outside top-128.”
“We depart from this recipe and, as in MTP boosting (Section 3.7), instead distill the full next-token distribution pM of the target model M”
“We use KL divergence as our objective.”
auxiliary loss 2
“We disable the main hard-target cross-entropy loss, MTP cross-entropy loss, and MoE load-balancing auxiliary loss and don’t use…”
“We mask out prompt, conversation-history, tool-output, and prefilled response-prefix tokens.”
optimization 22
optimizer 1
“Optimization uses distributed AdamW with β1 = 0.9, β2 = 0.95, ϵ = 10−8, weight decay 0.1, and gradient clipping at a global norm of 1.0.”
learning-rate schedule 1
“We reset the learning rate, warmed it up linearly to a maximum of1× 10−5, about twice the final pretraining learning rate, and then held…”
training precision 7
“The pre-training phase used an NVFP4 recipe.”
“The pre-training phase used an NVFP4 recipe.”
“we kept the QKV and attention projections in BF16”
“two optimizations designed to mitigate train–inference mismatch: router replay (Ma et al., 2025) and using FP32 for LM head (Chen et al.…”
“We kept the last 15% of the network in high precision to maintain stability.”
“To prevent loss of information, we keep these layers in MXFP8.”
“The NVFP4 format features: fine-grained (16 element) micro-block scaling, block scaling factors with E4M3 format, a second-level FP32…”
quantization-aware training 6
“such as randomness generation, stochastic rounding, or scaling calculations.”
“Activation scales are learned from a calibration set of image–text pairs, rather than text alone.”
“All other parameters remain frozen, including token embeddings, unquantized language-model parameters, normalization parameters, the…”
“We demonstrate stable and accurate pretraining on a hybrid Mamba-MoE architecture for up to 25T tokens using the NVFP4 number format.”
“Random Hadamard Transforms (RHTs) on inputs to wgrad”
“The released checkpoint uses the predefined aggressive PTQ recipe followed by SFT-based QAD at 200 iterations.”
training stability 2
“gradient clipping at a global norm of 1.0.”
“augmented with two optimizations designed to mitigate train–inference mismatch: router replay (Ma et al., 2025) and using FP32 for LM head”
training parallelism 3
“We used 16-way context parallelism, 8-way tensor parallelism, and 64-way expert parallelism to train on GB300 GPUs.”
“We used 8-way context parallelism, 8-way tensor parallelism, 8-way expert parallelism, and 2-way pipeline parallelism to train on GB200…”
“Pipeline parallelism left only one GPU active at a time, so we chose DP× EP.”
2 more mentioned or evaluated
training precision “Runaway moves by ≈ 0; output verbosity falls on 5 of 5 focused benchmarks”
training precision “No recovery of termination is captured.”
data sourcing 1
“The GitHub Crawl was collected using the GitHub REST API and the Amazon S3 API.”
data filtering 18
“We therefore performed checks of post-training data against 30 benchmarks (see the complete list in Appendix C.2) and removed samples…”
“we applied just heuristic filtering instead—similar to what we did for lower quality English data in the Nemotron-CC pipeline”
“we apply targeted keyword- and regex-based filters and remove all trajectories matching such behavior.”
“We then aggressively filter reasoning traces exhibiting pathological repetition, such as repeated n-grams within a sliding window or…”
“We collect raw source code and subsequently remove any having a license which does not exist in our permissive-license set.”
“we apply a unified data filtering pipeline to ensure that only high-quality, license-compliant, and verifiable samples are used for…”
“we apply automated quality checks to filter ambiguous or visually unsupported questions, incomplete reasoning traces, hallucinated labels…”
“Every source is decontaminated against SWE-bench and Terminal-Bench and has secrets removed.”
“retaining samples that are nearly impossible to solve correctly without using the available tools.”
“we removed reasoning responses shorter than 5,000 tokens to focus the training mixture on examples containing sufficiently developed…”
“Verbatim overlap measures how many n-grams of the question’s text appear in the sample.”
“The sample is reduced to its problem statement, the part into which a benchmark question could leak. Text templates that recur across its…”
“We retain normally completed trajectories whose answers match the references and whose required tools execute successfully.”
“per question: top 20 by combined score+top 2 per signal”
“We audit reference-answer formats and compatibility with the reward functions, and remove exact question–answer duplicates from the…”
“A sample is removed if the judge finds it severely contaminated with respect to any benchmark question.”
“We first discard malformed examples using structural checks (e.g., missing tool definitions when tool calls are present).”
“We keep translations scoring at least 75.”
deduplication 2
“Deduplication was done in the same way as for Nemotron-CC.”
“remove exact question–answer duplicates from the second-stage mixture.”
synthetic data 18
“we generate synthetic data by distilling trajectories, solutions, and translations from strong teacher models and agent systems, often…”
“We use Kimi K2.6 (Kimi Team, 2025) and K3 (Kimi Team, 2026) to generate agentic datasets with a focus on image tool operations such as…”
“For the ten additional lower-resource languages, we use Bodhan-MT for translation and Gemma-4-31B-it for quality assessment, with…”
“we synthetically generate long-context translation data from both the PG19 (Rae et al., 2019) and FinePDFs-Edu (Kydlíček et al., 2025)…”
“we improved scheduling so translation and quality checks can run concurrently.”
“The agentic track targets gaps found in a coverage audit against the Vals.ai category taxonomy by making the relationship among source…”
“The document-grounded track samples windowed contexts from SEC 10-K, 10-Q, and 8-K filings of S&P 500 companies, generates analyst-style…”
“We implement the four-stage pipeline for the twelve priority languages”
“Built with large-scale synthetic data generation, filtering, and curation”
“We use overlapping windows only for longer Santali documents”
“Lower-scoring translations are rewritten using the source text and identified errors, then checked again and kept only if they pass.”
“The model then checks each translation against its source and assigns a quality score from 0 to 100.”
“organizing the reasoning traces into an explicit sequence of concise steps, with each step grounded in relevant visual evidence, helps…”
“we synthetically generated graph-structured data to teach the model to trace nodes and edges through reasoning, rather than relying solely…”
“We address these gaps by translating curated English documents into Indic languages.”
“Fully synthetic, privacy-preserving personas grounded in real-world demographic, geographic, and cultural distributions”
“long-translation performance was improved by also including synthetic transduction tasks (Willette et al., 2026) that reinforce position…”
“translate each document in full, without splitting”
data mixture & curriculum 6
“we train a vision tool-calling teacher on a curated mixture of internal data.”
“are routed to the SFT or RL pool according to their measured difficulty.”
“10% (10B tokens) allocated to selected synthetic text from our corpus and natural Indic text.”
“Most of it is natural data kept in its original format, without chat templates.”
“We replace the broad RLVR mixture with a reasoning-focused curriculum of 25,600 examples, with each 256-prompt batch containing 90%…”
“stage 1 is weighted toward the domains that improve quickly and tend to regress later, and stage 2 toward the domains that improve more…”
3 more mentioned or evaluated
data mixture & curriculum “we ran separate Lightning 3.5 research experiments with 10%, 20%, and 30% Indic data under a fixed 200B-token training budget.”
data mixture & curriculum “a mixture of 75% SFT data with 25% teacher-generated RL rollouts”
synthetic data “mitigation strategies like counterfactual data augmentation to align with the desired model behavior”
filed at the root 1
“We further train specialized teachers for video reasoning and tracking, vision tool use, and alignment”
supervised fine-tuning 9
“The model was further fine-tuned on synthetic code, math, science, tool calling, instruction following, structured outputs, and general…”
“Establish a baseline, train and deploy a LoRA adapter, and evaluate task-specific improvements in minutes”
“we use a two-stage SFT process because learning vision and text capabilities simultaneously is challenging.”
“we do a small second stage of SFT, where we change the blend composition to boost some capabilities the first stage did not learn…”
“The agent traces are the only SFT-style data, so we keep them to a small share of the blend and mask tool responses from the loss…”
“the reasoning traces of a small fraction of SFT data are truncated to random reasoning budgets while the responses remain the same and…”
“Nemotron 3.5 Lightning is trained with single SFT stage with the text only SFT data.”
“supervised fine-tuning (SFT) was performed at a 256k sequence length”
“we reduced checkpointing interval to 100 iterations, used the 100% SFT blend only, and evaluated QAD checkpoints at 200, 300, and 400…”
reinforcement learning algorithm 8
“The model underwent multi-environment reinforcement learning using GRPO (Group Relative Policy Optimization) across math, code, science…”
“We post-train Nemotron 3.5 Super through a comprehensive pipeline spanning Supervised Fine Tuning (SFT), Reinforcement Learning from…”
“We initialize from an existing multimodal RLVR checkpoint and train using asynchronous GRPO”
“We mix this RLHF training with the abstention training strategy from Nemotron 3 Ultra”
“tokens are then importance-weighted by¯𝜋𝑆/¯𝜋𝑆−1 and those with weights outside[0.2, 5]are masked”
“continue training with full end to end GRPO on agentic tasks.”
“we use GRPO (Shao et al., 2024) with masked importance sampling to account for discrepancies between the training and rollout policies”
“length-based adjustments are applied on the RL rewards for them.”
reward modelling 8
“Use reinforcement learning with verifiable rewards to adapt NVIDIA Nemotron for specific domains and workflows”
“First, we train a new variant of the Nemotron 3 Ultra GenRM that is trained with a reasoning length penalty to be more token efficient.”
“as well as an auxiliary tool-use reward to penalize duplicate and malformed tool calls.”
“Video-QA uses a binary reward based on multiple-choice answer correctness.”
“the verifier assigns a continuous reward based on localization quality, object-visibility accuracy, and the validity and completeness of…”
“Second, we apply a soft length preference only to correct responses.”
“Both sources use deterministic, verifiable task rewards rather than an LLM judge.”
“a response without exactly one properly matched <think>...</think> block is considered malformed: its task reward is set to zero and its…”
preference optimization 2
“We train this teacher primarily using RLHF.”
“we instead compare all policy responses to a fixed response from a strong open-source model, Gemma 4 31B-it.”
policy distillation 6
“using the token-local update”
“After MOPD, truncation decreases to 0.3% with 13,669 output tokens per rollout, where only one runaway loop failure is captured.”
“then distilled each model for 1 trillion tokens”
“environments no specialist owns are routed to a self-anchor, a frozen copy of the student’s starting checkpoint”
“the distillation target is smoothed with the proximal-teacher construction of trust-region policy distillation”
“MOPD for Nemotron 3.5 Super is run in two stages on 128 GB200 nodes.”
rollout & RL infrastructure 5
“It utilized an asynchronous RL architecture that decouples training from inference and leverages MTP to accelerate rollout generation.”
“We generate on-policy rollouts using the corresponding backbone checkpoint.”
“For each task, N independent instances of the same post-trained agent attempt the task, producing a pool of candidate trajectories. These…”
“responses may come from the previous snapshot¯𝜋𝑆−1, i.e. one step stale”
“For the RL rollouts we sample 16 trajectories per source record with vLLM.”
agentic post-training 4
“we train the Nemotron 3 models on all of these tasks simultaneously”
“The environment provides the same tools—SEC filing search, document parsing, evidence retrieval, and final source-supported…”
“Harness-optimized training: The model is trained for popular agent harnesses, enabling agents to make more accurate calls while reducing…”
“opting to train the end to end agentic teacher on sandbox, fully executable agentic Terminal, SWE, and Search tasks.”
mid-training & continual pretraining 5
“In the LC-Phase, we performed continued pretraining (CPT) at 1M-token context length to equip the base model with long-context ability.”
“Mid-training sits between the two. It trains the base model at long context on a diverse mixture of natural software-engineering data with…”
“combines synthetic text generated from English documents with natural Indic corpora to strengthen Indian-language capabilities”
“To recover long-context capabilities affected by 256K mid-training, we perform an additional 9 billion token recovery stage”
“The mid-training mixture combines whole repositories, issue–pull-request pairs, commit histories, build and CI logs, and agent trajectories.”
2 more mentioned or evaluated
supervised fine-tuning “This SFT is an experimental agentic blend of about 33B tokens (2K steps), not the one used in Section 3”
supervised fine-tuning “fine-tuning with demographically balanced datasets”
filed at the root 1
“Rather than impose one serving configuration everywhere, we tune quantization and speculative decoding for each platform.”
decoding strategy 10
“This is a process whereby the model, or draft model, will draft some number of tokens which are efficiently reviewed.”
“LiLiCorr retains the K most probable tokens of the drafter body’s marginal as candidates”
“the system can speculate about: token t+1 token t+2 token t+3 ... Those candidates can then be verified efficiently.”
“candidate solutions, best-of-n selection traces, and verified CUDA kernels”
“parallel drafters propose all draft tokens in a single forward pass whose latency is nearly independent of the draft length.”
“LiLiCorr instead uses a lightweight, single-pass correlation head to score compatibility across slots.”
“It keeps the drafter body parallel and adds a lightweight sequential head that biases the logits at each slot using a low-rank transition…”
“DSpark: A semi-autoregressive speculative-decoding drafter that proposes a whole block of candidate tokens in a single forward pass from a…”
“DFlash: A speculative-decoding drafter that uses a lightweight block-diffusion model to generate an entire draft block in one forward pass.”
“Under probabilistic draft sampling, each slot is instead drawn from the head’s distribution over its candidates at the request’s…”
reasoning control 1
“the Nemotron 3 models are trained to work with inference-time budget control”
KV cache management 4
“We additionally optimize hybrid-state caching and GPU kernels for efficient long-context and agentic serving”
“The adapted path arranges for a logical chunk to end at the desired cache boundary, extracts the state and convolution window there, and…”
“We have since implemented a "checkpointing replay" method.”
“We improved this path by optionally retaining an additional Mamba cache page near the speculative-decoding replay boundary.”
inference quantization 20
“MTP KV precision therefore follows the backbone.”
“thus a single FP8 lm_head serves both target verification and MTP drafting”
“We apply post-training quantization (PTQ) and quantization-aware distillation (QAD) with ModelOpt on Lightning 3.5 and Super 3.5 VL”
“Hence we use the MSE calibration for the PTQ and AutoQuant recipes.”
“The released checkpoint uses the aggressive recipe, with NVFP4 MoE and shared-expert GEMMs and FP8 attention projections and Mamba linear…”
“The MTP attention weights remain BF16, with only the K/V BMM quantizers at FP8.”
“To maintain token generation verbosity, the lm_head should stay at FP8 weights or above”
“the mixed-precision checkpoints of §4.3, produced by a gradient-based per-layer search”
“We develop NVFP4 post-training quantization and mixed-precision recipes across weights, caches, and the MTP drafter”
“We therefore quantize the MTP experts using NVFP4 W4A4.”
“quantizing LiLiCorr’s linear layers to NVFP4 while keeping the activations at BF16, resulted in a 11% throughput improvement on DGX Spark”
“To address this issue, we preferred parameter alignment as described in §4.4.”
“This study recommends FP8 precision on 𝑞/𝑘/𝑣 projection at per-block scale granularity.”
“We use the SFT data employed during post-training as the calibration corpus and calibrate the model using 1,024 samples.”
“complemented by quantization-aware distillation, achieving 99.62% median accuracy recovery relative to BF16”
“The released checkpoints were produced by quantizing the boosted MTP separately and splicing it into the selected quantized backbone”
“AutoQuant estimates a sensitivity Sᵢ(q) from the quantization-induced output error weighted by the squared task-loss gradient.”
“we applied weight-only PTQ to improve the drafting efficiency of a DSpark module for Nemotron Lightning 3.5 and LiLiCorr module for…”
“Nemotron 3.5 Lightning ships with an NVFP4 checkpoint alongside BF16, using the same specialized NVFP4 kernels that power Nemotron 3 Ultra…”
“This allows the target lm_head to remain in BF16 while using a lower-precision lm_head for drafting.”
inference scheduling 2
“A KV-cache-aware router sends each request to an instance.”
“Session affinity sends all turns of one session to the same worker, so later turns find their prefix in its cache.”
inference kernel 3
“optimize hybrid-state caching and GPU kernels for efficient long-context and agentic serving”
“The multi-wave KV split then splits the KV cache further and runs the work in several waves.”
“The global linear backend remains the default, while a kernel configuration can override it for a specific quantization scheme”
agentic scaffolding 3
“Plans route up to the frontier, execution routes down to Lightning, ensuring that your tokens are spent efficiently and effectively.”
“We further improve the final model’s agentic performance through parallel agents followed by self-verification.”
“translates natural language requests into safe, human-approved terminal commands using tool calling, execution feedback, and optional…”
20 more mentioned or evaluated
serving parallelism “The disaggregated deployment uses 7 GPUs for prefill only and 4 GPUs for decode only.”
reasoning control “we explore test-time scaling through parallel agent execution and self-verification, improving Terminal-Bench 2.1 from 62.7% to 71.9%”
inference quantization “aggregation benchmarks can identify important candidates in FP8, where even per-tensor FP8 precision can be acceptable over 100 tasks at…”
inference quantization “Expert activations → BF16. Runaway moves by ≈ 0”
inference quantization “NVFP4 stores 4-bit E2M1 values with one FP8 E4M3 scale per 16-element block”
decoding strategy “MTP is best suited for medium to high concurrency, with the optimal draft length decreasing as concurrency increases.”
inference quantization “quantizing its weights to FP4 inflates generation length.”
inference quantization “in each case both the key and value caches across attention layers are quantized in FP8 or NVFP4, respectively.”
inference quantization “Empirically, promoting MoE expert precision and fine-tuning on RL rollouts can mitigate this effect”
inference quantization “with a linear E2M1 residual the KL falls to 0.00117 against per-tensor FP8’s 0.00134”
inference quantization “Substituting MXFP8 for per-tensor FP8 on 𝑞/𝑘/𝑣 changes only the scale granularity with one E8M0 exponent per 32 elements instead of one…”
inference quantization “Routed/shared experts NVFP4→ FP8. In tau3-banking, runaway dropped from 28.2% → 17.9%”
inference quantization “Further quantizing the LM head improved throughput by another 5%.”
inference quantization “MSE 4-over-6: MSE-based scale selection restricted to a narrower search space.”
inference quantization “Rotation or higher precision can mitigate this effect in activation”
inference quantization “AutoQuant was explored as an alternative sensitivity-driven mixed-precision approach.”
inference quantization “we emulated two-pass FP4 weights, where a second channel corrects the residual left below the amax/24 flush line.”
inference quantization “output verbosity increased by 18.5% at 2.4𝜎, while the runaway rate and knowledge abstention were both unaffected.”
inference quantization “Applying rotation or higher activation precision can mitigate the flush.”
inference quantization “Applying rotation or higher activation precision can mitigate the flush.”
Not used 3
decoding strategy “The Lightning 3.5 release used the lightweight Markov correction but omitted this confidence head.”
inference quantization “Per-layer mixed FP8-key/NVFP4-value (or vice-versa) configurations are not evaluated in this comparison.”
inference quantization “This tensor analysis stops us from taking the Mamba output projection to W4A4.”
filed at the root 2
“Understands document semantics and extract text and tables elements with spatial grounding”
“Model routing and orchestration help make this division of labor more accessible. They are now available through NVIDIA NeMo Switchyard.”
inference engine 1
“TensorRT™-LLM is an open source library built to deliver high-performance, real-time inference optimization for large language models”
infrastructure service 2
“NeMo Guardrails, a flexible, open library for defining and enforcing enterprise AI policies in real time”
“It routes prompts to the most capable and efficient model for each step of an agent workflow automatically, based on configurable routing…”
1 more mentioned or evaluated
kernel & quantization library “We use ModelOpt AutoQuantize (AutoQuant) to replace hand-authored layer rules with a sensitivity-constrained mixed-precision search.”
evaluation 33
filed at the root 11
“The benchmark side (question store, n-gram index, and embedding index) is built once and shared by all runs.”
“We evaluate MGSM (Shi et al., 2022) using 8-shot native chain-of-thought prompting in Spanish, German, French, Chinese, Japanese, and…”
“All models were evaluated under the same evaluation settings including agentic resources (CPU compute and timeouts, tools, search APIs)…”
“Judge gate semantic overlap≥0.75 or verbatim overlap≥0.5”
“It is important to stratify test cases along with capability partition to assess quantization damage properly.”
“Table 35 stratifies 1,470 MRCR-128k tasks at three repetitions to separate the retrieval from decoding noise.”
“As negatives we add unmodified questions from a different benchmark in the same domain, chosen to be hard to tell apart from the target”
“Chunking 512-byte windows with 256-byte stride (sample and question)”
“while tracking the failure of a model to terminate as it runs to the context wall”
“The LLM augmentations are generated with a model different from the judge, so that the judge cannot recognize its own rewrites.”
“transform each with 13 augmentations, at four strengths (0.25, 0.5, 0.75, 0.9)”
benchmark 5
“using EvalPlus-sanitized variants (Liu et al., 2023).”
“GPQA Diamond is evaluated using 5-shot chain-of-thought prompting.”
“MBPP-Sanitized uses 3-shot prompting”
“candidate solutions, best-of-n selection traces, and verified CUDA kernels”
“while HumanEval uses zero-shot prompting.”
evaluation harness 3
“All evaluation results reported for Nemotron 3.5 Super 120B-A12B Base and Lightning 3.5 30B-A3B Base were collected via NeMo Evaluator SDK”
“were collected via NeMo Evaluator SDK and NVIDIA’s open-source container of LM Evaluation Harness”
“NVIDIA says these evaluations were run through a consistent NeMo Gym and NeMo Evaluator-based harness”
judge 8
“Judge model Claude Opus 4.6, temperature 0”
“Passing both texts as JSON-escaped strings keeps their boundaries unambiguous even when the sample itself contains code, markup, or…”
“The same checkpoint then serves as a verifier, selecting one candidate using LLM-as-a-Verifier (LAV).”
“Each shortlisted pair of a sample and a benchmark question is verified by an LLM judge.”
“For each compared pair and criterion, we evaluate both presentation orders”
“These pairwise evaluations are organized using LAV’s Probabilistic Pivot Tournament (PPT).”
“other free-form answers require same-teacher, text-only equivalence judging with reported confidence≥ 0.90.”
“evaluates them against three criteria: specification adherence, output match, and error signal detection.”
averaging - mean@k; avg@k; pass@k; pass^k 3
“We additionally evaluate MATH-500 using 4-shot sampled generation with 32 generations per prompt and report pass@1.”
“We also use the mean of per rubric scores to evaluate responses instead of an overall score.”
“We report sampled pass@1 estimated from 32 generations per prompt.”
3 more mentioned or evaluated
benchmark “This index combines nine evaluations to measure model performance across agentic tasks, coding, scientific reasoning, and general…”
“To measure the effect of mid-training, we ran identical SFT on the base checkpoint and on the mid-trained checkpoint and compared the two.”
“Following the V-model methodology, iterative testing and validation at both unit and system levels are essential”
other 2
filed at the root 1
“We identify the runaway loop by reading reasoning passages in postmortem”
1 more mentioned or evaluated
“We divide and conquer the degradation by ablating non-attention components in pre-MOPD models”
unfiled 23
“processes the resulting (B−1)×K candidate lattice jointly with a small Transformer”
“use the expensive model to think and the fast model to work”
“using request signals and routing policies to send each agent task to the right model”
“convert Manipuri translations from Bengali script to Meitei Mayek”
“The sample is compared with every benchmark question using two signals.”
“Semantic overlap maximum cosine similarity over chunk pairs”
“Semantic overlap measures the embedding similarity between parts of the two texts.”
“generate executable SpatialClaw (Cho et al., 2026) demonstrations from existing video question–answer pairs, teaching the student to…”
“Frontier reasoning models such as Nemotron 3 Ultra handle orchestration and complex planning while smaller, more efficient models handle…”
“the block is committed from left to right, each slot selecting the candidate most compatible with its committed predecessor”
“The combination of these two quantities is a useful early-warning indicator.”
“truncation, the fraction of rollouts containing at least one model call hitting an output limit”
“NVIDIA exposes the model through an OpenAI-compatible endpoint.”
“We optimize only the weights of language-model GEMMs with enabled quantizers. All other parameters remain frozen”
“including multi-turn trajectories, tool calls, and preference signals across coding, math, reasoning, and agentic tasks”
“We replay the MoE expert-routing decisions recorded during rollout generation when evaluating the policy, reducing train–generation…”
“the evaluation recipes, installation instructions, and commands for NVIDIA Nemotron 3.5 Lightning were collected and published in NeMo Gym”
“The quantization configuration and calibrated quantizer statistics are restored from the PTQ checkpoint rather than recalibrated during QAD.”
“We optimize only the weights of language-model GEMMs with enabled quantizers.”
“Overall, our results favor shorter QAD training with 100% SFT data and show that optimal training duration depends on the selected data…”
“Speculative decoding, FP8 weights, and FlashAttention reduced median translation cost”
“Switchyard can expose Nemotron 3.5 Lightning as a routing target alongside your open and closed models, so every request lands on the most…”
1 more mentioned or evaluated
“we recommend considering evaluation techniques such as bias audits”
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- NVIDIA-Nemotron-3.5-Lightning-30B-A3B architecture card — LLM Architecture Gallery explainer sebastianraschka.com
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| developer.nvidia.com/blog/nvidia-nemotron-3-5-lightning-delivers-fast-a… | official blog | NVIDIA Technical Blog | read |
| arxiv.org/abs/2512.20856 | technical report | NVIDIA | read |
| research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Super-and-L… | technical report | NVIDIA | read |
| huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 | model card | NVIDIA | read |
| build.nvidia.com/nvidia/nemotron-3.5-lightning-30b-a3b/modelcard | vendor docs | NVIDIA | read |
| developer.nvidia.com/nemotron | vendor docs | NVIDIA | read |
| openrouter.ai/nvidia/nemotron-3.5-lightning | vendor docs | OpenRouter | read |
| analyticsvidhya.com/blog/2026/08/nvidia-nemotron-3-5-lightning | third party analysis | Analytics Vidhya | read |