Open Models Almanac
ModelsNVIDIA-Nemotron-3.5-Super-VL-120B-A12B

NVIDIA

NVIDIA-Nemotron-3.5-Super-VL-120B-A12B

What NVIDIA-Nemotron-3.5-Super-VL-120B-A12B’s own documents say it is built from — every method with the sentence that describes it.

Parameters
120B / 12B active
Context
1M
Attention
Mamba hybridhybrid Mamba-Transformer (nemotron_h_omni), RADIO vision encoder
Experts
LatentMoE, 512 routed + 1 shared, top-22
Input
text, image, video
Licence
OpenMDW-1.1
Released
2026-10-15

Against the consensus recipe

Its documents state 8 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 5 of the 19 architecture features checked — 3 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_nemotron_h_omni.py, configuration_nemotron_h_omni.py) and config.json at revision fb0ee0bc6b, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention code only in its codenot stated

The text_config sets 32 attention heads with only 2 key-value heads, indicating grouped-query attention in the attention layers, though…

show evidence

The text_config sets 32 attention heads with only 2 key-value heads, indicating grouped-query attention in the attention layers, though the nemotron_h attention implementation file itself was not provided.

llm_config.num_attention_heads = 32llm_config.num_key_value_heads = 2
Multi-head latent attention code unclearnot stated

No kv_lora_rank/q_lora_rank keys appear in the text_config and the nemotron_h attention code is not included, so MLA cannot be confirmed…

show the whole verdict

No kv_lora_rank/q_lora_rank keys appear in the text_config and the nemotron_h attention code is not included, so MLA cannot be confirmed or ruled out.

Sliding-window attention not in its codenot stated

sliding_window is explicitly null in the text_config, so no sliding-window restriction is applied even if supported elsewhere.

show evidencellm_config.sliding_window = null
Interleaved sliding-window and global attention not in its codenot stated

Since sliding_window is null, there is no interleaving of sliding and full attention; layer typing only alternates mamba/moe/attention…

show evidence

Since sliding_window is null, there is no interleaving of sliding and full attention; layer typing only alternates mamba/moe/attention, not windowed vs global attention.

llm_config.sliding_window = null
Indexer-selected sparse attention (DeepSeek Sparse Attention) not in its codenot stated

No index_topk, index_n_heads, or index_head_dim keys appear anywhere in the config, and no indexer code is present in the given files.

Linear-attention or state-space layers alongside full attention code unclearnot stated

layers_block_type interleaves 'mamba' (a linear-complexity SSM mixer) with 'moe' feed-forward blocks and occasional 'attention' softmax…

show the whole verdict

layers_block_type interleaves 'mamba' (a linear-complexity SSM mixer) with 'moe' feed-forward blocks and occasional 'attention' softmax layers in the text_config.

Gated DeltaNet layers code unclearnot stated

The config has mamba-specific keys (mamba_num_heads, conv_kernel, ssm_state_size) rather than linear_num_key_heads/linear_conv_kernel_dim…

show the whole verdict

The config has mamba-specific keys (mamba_num_heads, conv_kernel, ssm_state_size) rather than linear_num_key_heads/linear_conv_kernel_dim used by Gated DeltaNet, and the mixer implementation file is not provided, so it is unclear whether the mamba layers use the gated delta rule versus standard Mamba-2 SSD.

Mamba-2 layers code only in its codenot stated

The text_config declares Mamba-2-style SSM hyperparameters (ssm_state_size, mamba_num_heads, mamba_head_dim, chunk_size) used by the…

show evidence

The text_config declares Mamba-2-style SSM hyperparameters (ssm_state_size, mamba_num_heads, mamba_head_dim, chunk_size) used by the 'mamba' layers in layers_block_type, matching NemotronH's Mamba-2 mixer design.

llm_config.ssm_state_size = 128llm_config.mamba_num_heads = 128llm_config.mamba_head_dim = 64llm_config.chunk_size = 128
Learnable attention sink code unclearnot stated

No attention-sink related keys or code are present in the given config/files to confirm or rule out a learned sink logit in the attention…

show the whole verdict

No attention-sink related keys or code are present in the given config/files to confirm or rule out a learned sink logit in the attention layers.

Gated attention output code unclearnot stated

No attn_output_gate/use_output_gate keys appear and the nemotron_h attention implementation is not included, so an input-dependent sigmoid…

show the whole verdict

No attn_output_gate/use_output_gate keys appear and the nemotron_h attention implementation is not included, so an input-dependent sigmoid output gate cannot be confirmed.

QK normalization code unclearnot stated

No use_qk_norm/qk_layernorm keys appear in the config and the attention implementation file is not provided, so QK normalization cannot be…

show the whole verdict

No use_qk_norm/qk_layernorm keys appear in the config and the attention implementation file is not provided, so QK normalization cannot be confirmed.

Partial RoPE code unclearnot stated

No partial_rotary_factor, qk_rope_head_dim, or rotary_dim keys appear in the text_config, and the attention/rope code is not included, so…

show the whole verdict

No partial_rotary_factor, qk_rope_head_dim, or rotary_dim keys appear in the text_config, and the attention/rope code is not included, so partial RoPE cannot be determined.

YaRN RoPE scaling not in its codenot stated

No rope_scaling or rope_parameters key is present anywhere in the config, so no YaRN scaling is configured.

Layers without positional encoding (NoPE) code unclearnot stated

No no_rope_layers or nope_layer_interval keys appear in the config, and the attention implementation deciding RoPE application per layer…

show the whole verdict

No no_rope_layers or nope_layer_interval keys appear in the config, and the attention implementation deciding RoPE application per layer is not provided.

Mixture of experts in its codeadopted

The text_config defines 512 routed experts with 22 selected per token, used by the 'moe' blocks in layers_block_type.

show evidencellm_config.n_routed_experts = 512llm_config.num_experts_per_tok = 22
Shared expert code only in its codenot stated

The text_config declares 1 shared expert with a nonzero intermediate size, applied alongside the routed experts in MoE layers.

show evidencellm_config.n_shared_experts = 1llm_config.moe_shared_expert_intermediate_size = 5376
Auxiliary-loss-free load balancing (selection bias) code unclearnot stated

No topk_method or e_score_correction_bias key is present in the config, and the MoE router code is not included, so bias-based load…

show the whole verdict

No topk_method or e_score_correction_bias key is present in the config, and the MoE router code is not included, so bias-based load balancing cannot be confirmed.

Multi-token prediction layers in its codeadopted

The text_config declares num_nextn_predict_layers=1 and an mtp_layers_block_type, and the omni modeling file conditionally builds an extra…

show evidence

The text_config declares num_nextn_predict_layers=1 and an mtp_layers_block_type, and the omni modeling file conditionally builds an extra vision norm specifically for checkpoints whose language model carries MTP layers, confirming MTP modules are active for this config.

llm_config.num_nextn_predict_layers = 1
            nn.LayerNorm(config.vision_hidden_size, eps=config.vision_config.layer_norm_eps)
            if config.text_config.num_nextn_predict_layers > 0
            else None
modeling_nemotron_h_omni.py · L157–159
SwiGLU feed-forward not in its codenot stated

The text_config sets mlp_hidden_act to 'relu2' rather than a SiLU-gated activation, so the feed-forward/expert MLPs are not SwiGLU.

show evidencellm_config.mlp_hidden_act = "relu2"

filed at the root 1

Hybrid architecture core

“The model builds on a strong foundation and uses a hybrid architecture, speculative decoding, context reuse, and mixed precision”

token mixer 4

Grouped dynamic convolution core

“each of the five layers of the drafter body wraps the attention and MLP sublayers in a grouped dynamic convolution”

softmax attention 1

sliding window attention 1

Sliding Window Attention with an 8,192-Position Window used

“We therefore serve it with sliding-window attention over the most recent 8,192 positions in every drafter layer”

linear attention & state space 1

Mamba 1

Mamba checkpointing core

“Hybrid-state caching, prefix reuse, Mamba checkpointing, GPU-kernel selection, and speculative decoding are jointly tuned”

hybrid layer stacking 1

Hybrid Mamba-Transformer core

“both models combine large-scale distillation, long-context training, and native speculative decoding”

channel mixer 2

mixture of experts 2

Mixture of experts core

“Its 120B-parameter mixture-of-experts (MoE) architecture activates 12B parameters per token”

expert load balancing 1

Disabled router-bias updates used

“Router-bias updates are explicitly disabled.”

prediction head 8

Multi-Token Prediction core

“Multi-token prediction (MTP) gives Nemotron 3.5 Super VL an internal drafter that proposes several tokens for the main model to verify in…”

Frozen-backbone MTP-head-only fine-tuning core

“the backbone remains frozen and only the MTP head parameters are updated.”

Shared Multi-Token Prediction head core

“A shared MTP head predicts future tokens recursively for speculative decoding.”

Recursively shared Multi-Token Prediction head core

“adopting the recursively shared MTP formulation”

Hidden-state mixing during recursive MTP training used

“input hidden states are sampled from the sequence of states produced by earlier MTP steps 1,...,k − 1, rather than relying exclusively on…”

Architecture-aware MTP-head initialization by projection fusion and pruning used

“We first fuse the latent MoE projections into the experts’ up- and down-projection matrices, then prune the hidden size from 4096 to 2688”

MTP-head warm-up with a frozen backbone used

“conduct a dedicated MTP warmup phase for 100 B tokens, freezing the backbone parameters and optimizing only the MTP head”

MTP-only training used

“We apply a dedicated MTP-only training recipe at two points”

multimodal architecture 3

C-RADIO vision encoder core

“The vision encoder and connector turn visual inputs into representations processed by the language backbone.”

Vision-language connector core

“The vision encoder and connector turn visual inputs into representations processed by the language backbone.”

Video-frame pruning used

“the video configuration samples up to 256 frames at 2 FPS with a pruning rate of 0.5.”

filed at the root 1

Target-distribution calibration loss used

“We replace it with a calibration term.”

multi-token prediction objective 3

MTP Boosting used

“A dedicated MTP-boosting training phase further aligned the drafter with the final model.”

LK loss used

“Whereas Nemotron 3 Ultra uses a temperature-scaled forward-KL objective, we now use the LK loss”

MTP warm-up used

“Both models begin with a dedicated 100-billion-token MTP warm-up”

distillation objective 8

Reverse KL divergence core

“The student minimizes the reverse KL𝐷KL(𝜋𝑆(·|𝑐𝑡)‖𝜋𝑇 (·|𝑐𝑡))at every position of its own responses”

Quantization-aware distillation used

“The released checkpoint uses the predefined aggressive PTQ recipe followed by SFT-based QAD at 200 iterations.”

Skew reverse-KL distillation used

“with Ultra 3 Base as the teacher and Skew Reverse KL divergence as the distillation loss function”

Backbone-to-MTP-head knowledge distillation used

“these runs use a knowledge-distillation loss with the frozen backbone as the teacher.”

Combined KL-divergence and cross-entropy distillation loss used

“we combined the KL loss with cross-entropy loss, assigning them weights of 0.9 and 0.1, respectively”

Forward-KL distillation with a sink token used

“We used standard forward KL divergence loss along with an extra sink token to capture the residual probability mass outside top-128.”

Full next-token distribution distillation used

“We depart from this recipe and, as in MTP boosting (Section 3.7), instead distill the full next-token distribution pM of the target model M”

KL-divergence distillation objective used

“We use KL divergence as our objective.”

auxiliary loss 2

Disable auxiliary and hard-target losses during QAD used

“We disable the main hard-target cross-entropy loss, MTP cross-entropy loss, and MoE load-balancing auxiliary loss and don’t use…”

Mask non-target tokens in rollout training data used

“We mask out prompt, conversation-history, tool-output, and prefilled response-prefix tokens.”

optimizer 1

Distributed AdamW used

“Optimization uses distributed AdamW with β1 = 0.9, β2 = 0.95, ϵ = 10−8, weight decay 0.1, and gradient clipping at a global norm of 1.0.”

learning-rate schedule 1

Linear learning-rate warmup followed by a constant rate used

“We reset the learning rate, warmed it up linearly to a maximum of1× 10−5, about twice the final pretraining learning rate, and then held…”

training precision 2

NVFP4 core

“It uses NVFP4 quantization to reduce memory requirements”

FP32 language-model head used

“two optimizations designed to mitigate train–inference mismatch: router replay (Ma et al., 2025) and using FP32 for LM head (Chen et al.…”

quantization-aware training 4

Stochastic Rounding used

“such as randomness generation, stochastic rounding, or scaling calculations.”

Calibration Data used

“Activation scales are learned from a calibration set of image–text pairs, rather than text alone.”

Freezing Non-Target Parameters used

“All other parameters remain frozen, including token embeddings, unquantized language-model parameters, normalization parameters, the…”

SFT-Based Quantization-Aware Distillation used

“The released checkpoint uses the predefined aggressive PTQ recipe followed by SFT-based QAD at 200 iterations.”

training stability 2

Global-norm gradient clipping used

“gradient clipping at a global norm of 1.0.”

Router replay used

“augmented with two optimizations designed to mitigate train–inference mismatch: router replay (Ma et al., 2025) and using FP32 for LM head”

training parallelism 3

Combined context, tensor, and expert parallel training used

“We used 16-way context parallelism, 8-way tensor parallelism, and 64-way expert parallelism to train on GB300 GPUs.”

Combined context, tensor, expert, and pipeline parallel training used

“We used 8-way context parallelism, 8-way tensor parallelism, 8-way expert parallelism, and 2-way pipeline parallelism to train on GB200…”

Data and expert parallelism for calibration used

“Pipeline parallelism left only one GPU active at a time, so we chose DP× EP.”

2 more mentioned or evaluated
BF16 expert activations evaluated

training precision “Runaway moves by ≈ 0; output verbosity falls on 5 of 5 focused benchmarks”

BF16 language-model head evaluated

training precision “No recovery of termination is captured.”

data filtering 15

Benchmark decontamination used

“We therefore performed checks of post-training data against 30 benchmarks (see the complete list in Appendix C.2) and removed samples…”

Heuristic filtering used

“As we did not have reliable multilingual model-based quality classifiers available, we applied just heuristic filtering instead”

Permissive-license filtering used

“We collect raw source code and subsequently remove any having a license which does not exist in our permissive-license set.”

Automated quality filtering of synthetic vision data used

“we apply automated quality checks to filter ambiguous or visually unsupported questions, incomplete reasoning traces, hallucinated labels…”

Benchmark decontamination and secret removal from training sources used

“Every source is decontaminated against SWE-bench and Terminal-Bench and has secrets removed.”

Difficulty filtering for tool-dependent training examples used

“retaining samples that are nearly impossible to solve correctly without using the available tools.”

Minimum-length filtering for reasoning SFT data used

“we removed reasoning responses shorter than 5,000 tokens to focus the training mixture on examples containing sufficiently developed…”

N-gram verbatim-overlap detection used

“Verbatim overlap measures how many n-grams of the question’s text appear in the sample.”

Normalize training samples to their problem statements used

“The sample is reduced to its problem statement, the part into which a benchmark question could leak. Text templates that recur across its…”

Outcome- and execution-based filtering of agentic trajectories used

“We retain normally completed trajectories whose answers match the references and whose required tools execute successfully.”

Per-question retrieval of contamination candidates used

“per question: top 20 by combined score+top 2 per signal”

Reference-answer compatibility auditing used

“We audit reference-answer formats and compatibility with the reward functions, and remove exact question–answer duplicates from the…”

Severity-thresholded benchmark contamination removal used

“A sample is removed if the judge finds it severely contaminated with respect to any benchmark question.”

Synthetic trajectory filtering for correctness, repetition, and validity used

“Synthetic trajectories are filtered for structural correctness, repetition, and task validity.”

Translation quality-score threshold filtering used

“We keep translations scoring at least 75.”

deduplication 2

Corpus deduplication used

“The data was subsequently deduplicated and filtered in various ways used in the Nemotron-CC data pipeline.”

Exact question–answer deduplication used

“remove exact question–answer duplicates from the second-stage mixture.”

synthetic data 17

Agentic visual tool use for image operations used

“We use Kimi K2.6 (Kimi Team, 2025) and K3 (Kimi Team, 2026) to generate agentic datasets with a focus on image tool operations such as…”

Bodhan-MT translation with Gemma-4 quality assessment and language-specific thresholds used

“For the ten additional lower-resource languages, we use Bodhan-MT for translation and Gemma-4-31B-it for quality assessment, with…”

Chunked synthetic long-context translation data used

“we synthetically generate long-context translation data from both the PG19 (Rae et al., 2019) and FinePDFs-Edu (Kydlíček et al., 2025)…”

Concurrent scheduling of translation and quality checks used

“we improved scheduling so translation and quality checks can run concurrently.”

Coverage-audit-driven agentic question generation used

“The agentic track targets gaps found in a coverage audit against the Vals.ai category taxonomy by making the relationship among source…”

Document-grounded synthetic question generation with answerability verification used

“The document-grounded track samples windowed contexts from SEC 10-K, 10-Q, and 8-K filings of S&P 500 companies, generates analyst-style…”

Filtering synthetic trajectories for correctness and validity used

“Synthetic trajectories are filtered for structural correctness, repetition, and task validity.”

Four-stage Indic synthetic-data generation pipeline used

“We implement the four-stage pipeline for the twelve priority languages”

Overlapping windows for longer Santali documents used

“We use overlapping windows only for longer Santali documents”

Rewrite low-scoring translations using source text and identified errors, then recheck used

“Lower-scoring translations are rewritten using the source text and identified errors, then checked again and kept only if they pass.”

Score translations against their source for quality used

“The model then checks each translation against its source and assigns a quality score from 0 to 100.”

Step-structured, visually grounded reasoning traces used

“organizing the reasoning traces into an explicit sequence of concise steps, with each step grounded in relevant visual evidence, helps…”

Synthetic data generation with NeMo Data Designer used

“Diverse training data generated with NeMo Data Designer helped the model better understand complex charts.”

Synthetic graph-structured long-context reasoning data used

“we synthetically generated graph-structured data to teach the model to trace nodes and edges through reasoning, rather than relying solely…”

Synthetic Indic training-data generation by translating curated English documents used

“We address these gaps by translating curated English documents into Indic languages.”

Synthetic transduction-task training for long translation used

“long-translation performance was improved by also including synthetic transduction tasks (Willette et al., 2026) that reinforce position…”

Translate each selected source document in full without splitting used

“translate each document in full, without splitting”

data mixture & curriculum 6

Curated internal-data mixture for vision tool-use training used

“we train a vision tool-calling teacher on a curated mixture of internal data.”

Difficulty-based routing of validated questions to SFT or RL used

“are routed to the SFT or RL pool according to their measured difficulty.”

Indic continued-pretraining data allocation used

“10% (10B tokens) allocated to selected synthetic text from our corpus and natural Indic text.”

Preserve natural training-data format used

“Most of it is natural data kept in its original format, without chat templates.”

Reasoning-focused RL curriculum used

“We replace the broad RLVR mixture with a reasoning-focused curriculum of 25,600 examples, with each 256-prompt batch containing 90%…”

Stage-specific domain weighting by distillation learning curves used

“stage 1 is weighted toward the domains that improve quickly and tend to regress later, and stage 2 toward the domains that improve more…”

sequence packing 1

Packed-sequence training for long contexts used

“Training uses packed sequences of up to 524,288 tokens.”

2 more mentioned or evaluated
Fixed-budget Indic mixture-ratio ablations evaluated

data mixture & curriculum “we ran separate Lightning 3.5 research experiments with 10%, 20%, and 30% Indic data under a fixed 200B-token training budget.”

Mix supervised fine-tuning data with teacher-generated reinforcement-learning rollouts evaluated

data mixture & curriculum “a mixture of 75% SFT data with 25% teacher-generated RL rollouts”

filed at the root 1

Specialized teacher training used

“We further train specialized teachers for video reasoning and tracking, vision tool use, and alignment”

supervised fine-tuning 10

Two-stage supervised fine-tuning core

“A two-stage SFT pipeline trains the language and vision components”

Supervised fine-tuning used

“Super’s post-training pipeline combines supervised fine-tuning”

Capability-targeted second-stage supervised fine-tuning used

“we do a small second stage of SFT, where we change the blend composition to boost some capabilities the first stage did not learn…”

Mask tool responses from the SFT loss used

“The agent traces are the only SFT-style data, so we keep them to a small share of the blend and mask tool responses from the loss…”

Post-training on organization-specific data used

“Dozens of organizations across industries are post-training Super VL on their own data, tools, policies and workflows”

Random-budget truncation of reasoning traces used

“the reasoning traces of a small fraction of SFT data are truncated to random reasoning budgets while the responses remain the same and…”

Single-stage supervised fine-tuning used

“Nemotron 3.5 Lightning is trained with single SFT stage with the text only SFT data.”

Use 100% SFT data for final QAD used

“we reduced checkpointing interval to 100 iterations, used the 100% SFT blend only, and evaluated QAD checkpoints at 200, 300, and 400…”

Low-Rank Adaptation optional

“Developers can use NVIDIA NeMo AutoModel for supervised fine-tuning (SFT) and low-rank adaptation (LoRA) to teach domain-specific behavior.”

Domain-specific post-training optional

“Developers can post-train Nemotron 3.5 Super VL with NVIDIA NeMo using their own data, tools, policies, and workflows.”

reinforcement learning algorithm 7

Reinforcement learning used

“then post-trained with roughly 3.8 million samples and reinforcement learning across dozens of environments.”

Reinforcement learning from verifiable rewards used

“We post-train Nemotron 3.5 Super through a comprehensive pipeline spanning Supervised Fine Tuning (SFT), Reinforcement Learning from…”

Asynchronous Group Relative Policy Optimization used

“We initialize from an existing multimodal RLVR checkpoint and train using asynchronous GRPO”

Abstention training used

“We mix this RLHF training with the abstention training strategy from Nemotron 3 Ultra”

Clipped-range importance weighting with stale-token masking used

“tokens are then importance-weighted by¯𝜋𝑆/¯𝜋𝑆−1 and those with weights outside[0.2, 5]are masked”

End-to-end GRPO on agentic tasks used

“continue training with full end to end GRPO on agentic tasks.”

Length-based RL reward adjustment used

“length-based adjustments are applied on the RL rewards for them.”

reward modelling 8

Reinforcement Learning with Verifiable Rewards used

“Reinforcement learning with verifiable rewards, reinforcement learning from human feedback, and specialized teacher training”

Reasoning-length penalty used

“First, we train a new variant of the Nemotron 3 Ultra GenRM that is trained with a reasoning length penalty to be more token efficient.”

Auxiliary tool-use reward used

“as well as an auxiliary tool-use reward to penalize duplicate and malformed tool calls.”

Binary answer-correctness reward used

“Video-QA uses a binary reward based on multiple-choice answer correctness.”

Continuous tracking-quality reward used

“the verifier assigns a continuous reward based on localization quality, object-visibility accuracy, and the validity and completeness of…”

Correct-response-only soft length preference used

“Second, we apply a soft length preference only to correct responses.”

Deterministic task-verifier rewards used

“Both sources use deterministic, verifiable task rewards rather than an LLM judge.”

Matched think-block format validation used

“a response without exactly one properly matched <think>...</think> block is considered malformed: its task reward is set to zero and its…”

preference optimization 2

Reinforcement Learning from Human Feedback (RLHF) core

“We train this teacher primarily using RLHF.”

Fixed-reference response comparison used

“we instead compare all policy responses to a fixed response from a strong open-source model, Gemma 4 31B-it.”

policy distillation 8

Token-Local Distillation Update core

“using the token-local update”

Multi-Teacher On-Policy Distillation used

“After MOPD, truncation decreases to 0.3% with 13,669 output tokens per rollout, where only one runaway loop failure is captured.”

Distillation from a Larger Nemotron 3 Model used

“then distilled each model for 1 trillion tokens”

Distillation from Nemotron 3 Ultra Base used

“followed by 1T tokens of distillation from Nemotron 3 Ultra Base.”

Frozen Student Self-Anchor used

“environments no specialist owns are routed to a self-anchor, a frozen copy of the student’s starting checkpoint”

Multi-Teacher Capability Consolidation used

“consolidating capabilities from reasoning, agentic, alignment, video, and vision-tool-use teachers”

Proximal-Teacher Trust-Region Policy Distillation used

“the distillation target is smoothed with the proximal-teacher construction of trust-region policy distillation”

Two-Stage Multi-Teacher On-Policy Distillation used

“MOPD for Nemotron 3.5 Super is run in two stages on 128 GB200 nodes.”

rollout & RL infrastructure 4

On-policy rollout generation used

“We generate on-policy rollouts using the corresponding backbone checkpoint.”

Independent parallel agent rollouts used

“For each task, N independent instances of the same post-trained agent attempt the task, producing a pool of candidate trajectories. These…”

One-step-stale asynchronous response generation used

“responses may come from the previous snapshot¯𝜋𝑆−1, i.e. one step stale”

Teacher-generated rollout trajectories used

“For the RL rollouts we sample 16 trajectories per source record with vLLM.”

agentic post-training 2

Finance-agent environment with consistent tools across training stages used

“The environment provides the same tools—SEC filing search, document parsing, evidence retrieval, and final source-supported…”

Training agentic teachers on sandboxed executable tasks used

“opting to train the end to end agentic teacher on sandbox, fully executable agentic Terminal, SWE, and Search tasks.”

mid-training & continual pretraining 6

Long-context extension via continued pretraining core

“In the LC-Phase, we performed continued pretraining (CPT) at 1M-token context length to equip the base model with long-context ability.”

Mid-training core

“Mid-training sits between the two. It trains the base model at long context on a diverse mixture of natural software-engineering data with…”

Indic continued pretraining used

“combines synthetic text generated from English documents with natural Indic corpora to strengthen Indian-language capabilities”

Long-context recovery training used

“To recover long-context capabilities affected by 256K mid-training, we perform an additional 9 billion token recovery stage”

Software-engineering mid-training used

“A 65.4-billion-token stage at 256K context uses repositories, issue–pull-request pairs, commit histories, build and CI logs, and agent…”

Continued pretraining with natural and synthetic data used

“Continued pretraining with high-quality natural and synthetic data improved Indic-language capabilities”

1 more mentioned or evaluated
Experimental agentic SFT blend evaluated

supervised fine-tuning “This SFT is an experimental agentic blend of about 33B tokens (2K steps), not the one used in Section 3”

filed at the root 1

Platform- and workload-specific quantization and speculative-decoding tuning used

“Rather than impose one serving configuration everywhere, we tune quantization and speculative decoding for each platform.”

decoding strategy 10

Speculative decoding core

“The model builds on a strong foundation and uses a hybrid architecture, speculative decoding, context reuse, and mixed precision”

Top-K candidate lattice core

“LiLiCorr retains the K most probable tokens of the drafter body’s marginal as candidates”

Standalone parallel speculative drafters used

“parallel drafters propose all draft tokens in a single forward pass whose latency is nearly independent of the draft length.”

LiLiCorr single-pass correlation head used

“LiLiCorr instead uses a lightweight, single-pass correlation head to score compatibility across slots.”

Semi-autoregressive drafting with sequential Markov correction used

“It keeps the drafter body parallel and adds a lightweight sequential head that biases the logits at each slot using a low-rank transition…”

DSpark optional

“DFlash and DSpark for Nemotron 3.5 Lightning”

DFlash optional

“DFlash and DSpark for Nemotron 3.5 Lightning”

LiLiCorr optional

“LiLiCorr improves parallel speculative decoding by jointly evaluating token candidates across an entire draft”

MTP draft-length configuration optional

“In vLLM, num_speculative_tokens controls draft length; in SGLang, the MTP draft length uses --speculative-num-steps to config.”

Probabilistic draft sampling optional

“Under probabilistic draft sampling, each slot is instead drawn from the head’s distribution over its candidates at the request’s…”

reasoning control 3

Reasoning enabled by default default

“Reasoning is on by default and can be turned off or given a token budget.”

Thinking mode optional

“Super supports reasoning-off, regular, and medium-effort modes”

Reasoning token-budget control optional

“The regular and medium-effort reasoning modes have the option to be used in conjunction with inference-time budget control.”

KV cache management 8

Prefix caching core

“Prefix caching reuses computation from those shared inputs.”

Hybrid-state caching core

“Hybrid-state caching, prefix reuse, Mamba checkpointing, GPU-kernel selection, and speculative decoding are jointly tuned”

Attention-cache and Mamba-state preservation core

“The model’s serving optimizations preserve both attention caches and Mamba recurrent states.”

Prefix reuse core

“Hybrid-state caching, prefix reuse, Mamba checkpointing, GPU-kernel selection, and speculative decoding are jointly tuned”

Internal prefill checkpointing used

“The adapted path arranges for a logical chunk to end at the desired cache boundary, extracts the state and convolution window there, and…”

Mamba checkpointing for speculative-decoding replay used

“Additional Mamba checkpoints reduce the replay needed to recover inputs for speculative decoding”

Checkpointing replay for Mamba recurrent-state cache used

“We have since implemented a "checkpointing replay" method.”

KV-cache management optional

“NVIDIA Dynamo supports distributed inference optimizations to increase throughput and reduce latency, including KV-cache-aware routing, KV…”

inference quantization 24

Backbone-coupled MTP KV-cache precision core

“MTP KV precision therefore follows the backbone.”

Mixed-precision inference core

“The model builds on a strong foundation and uses a hybrid architecture, speculative decoding, context reuse, and mixed precision”

Native 2-bit quantization core

“using a native 2-bit quantization to fit in 24GB of memory”

Shared FP8 lm_head for target verification and MTP drafting default

“thus a single FP8 lm_head serves both target verification and MTP drafting”

Post-training quantization used

“We apply post-training quantization (PTQ) and quantization-aware distillation (QAD) with ModelOpt on Lightning 3.5 and Super 3.5 VL”

MSE-based weight-scale selection used

“Hence we use the MSE calibration for the PTQ and AutoQuant recipes.”

Aggressive mixed-precision quantization recipe used

“The released checkpoint uses the aggressive recipe, with NVFP4 MoE and shared-expert GEMMs and FP8 attention projections and Mamba linear…”

BF16 MTP attention weights with FP8 K/V BMM quantizers used

“The MTP attention weights remain BF16, with only the K/V BMM quantizers at FP8.”

Component-wise precision selection used

“Precision is selected separately for sensitive components.”

FP8-or-higher lm_head precision used

“To maintain token generation verbosity, the lm_head should stay at FP8 weights or above”

Gradient-based per-layer mixed-precision quantization search used

“the mixed-precision checkpoints of §4.3, produced by a gradient-based per-layer search”

NVFP4 post-training quantization used

“We develop NVFP4 post-training quantization and mixed-precision recipes across weights, caches, and the MTP drafter”

NVFP4 W4A4 quantization of MTP experts used

“We therefore quantize the MTP experts using NVFP4 W4A4.”

NVFP4 weight quantization with BF16 activations used

“quantizing LiLiCorr’s linear layers to NVFP4 while keeping the activations at BF16, resulted in a 11% throughput improvement on DGX Spark”

Parameter alignment for quantization used

“To address this issue, we preferred parameter alignment as described in §4.4.”

Per-block FP8 quantization for attention Q/K/V projections used

“This study recommends FP8 precision on 𝑞/𝑘/𝑣 projection at per-block scale granularity.”

Platform-dependent W4A4/W4A16 quantization used

“Inference uses W4A4 on Blackwell and W4A16 on non-Blackwell platforms”

Post-training calibration sample count used

“We use the SFT data employed during post-training as the calibration corpus and calibrate the model using 1,024 samples.”

Quantization-aware distillation used

“complemented by quantization-aware distillation, achieving 99.62% median accuracy recovery relative to BF16”

Separate MTP quantization and backbone splicing used

“The released checkpoints were produced by quantizing the boosted MTP separately and splicing it into the selected quantized backbone”

Task-loss-gradient-weighted quantization sensitivity estimation used

“AutoQuant estimates a sensitivity Sᵢ(q) from the quantization-induced output error weighted by the squared task-loss gradient.”

Weight-only post-training quantization for parallel drafters used

“we applied weight-only PTQ to improve the drafting efficiency of a DSpark module for Nemotron Lightning 3.5 and LiLiCorr module for…”

NVFP4 weight-and-activation quantization optional

“The NVFP4 recipe uses 4-bit weights and activations on the NVIDIA Blackwell architecture.”

Separate target and MTP lm_head precision optional

“This allows the target lm_head to remain in BF16 while using a lower-precision lm_head for drafting.”

serving parallelism 1

Disaggregated serving optional

“NVIDIA Dynamo supports distributed inference optimizations to increase throughput and reduce latency, including KV-cache-aware routing, KV…”

inference scheduling 4

KV-cache-aware routing used

“A KV-cache-aware router sends each request to an instance.”

Session affinity routing used

“Session affinity sends all turns of one session to the same worker, so later turns find their prefix in its cache.”

Asynchronous inference and execution optional

“--async-scheduling”

Provider failover routing optional

“we can recover by routing to another healthy provider, if your request filters allow it”

inference kernel 4

GPU-kernel selection core

“Hybrid-state caching, prefix reuse, Mamba checkpointing, GPU-kernel selection, and speculative decoding are jointly tuned”

GPU-kernel optimization used

“optimize hybrid-state caching and GPU kernels for efficient long-context and agentic serving”

Multi-wave KV-cache splitting optional

“The multi-wave KV split then splits the KV cache further and runs the work in several waves.”

Per-quantization-scheme GEMM backend selection optional

“The global linear backend remains the default, while a kernel configuration can override it for a specific quantization scheme”

context management 1

Context reuse core

“The model builds on a strong foundation and uses a hybrid architecture, speculative decoding, context reuse, and mixed precision”

agentic scaffolding 1

Parallel agent execution with self-verification used

“We further improve the final model’s agentic performance through parallel agents followed by self-verification.”

19 more mentioned or evaluated
Parallel test-time compute scaling evaluated

reasoning control “we explore test-time scaling through parallel agent execution and self-verification, improving Terminal-Bench 2.1 from 62.7% to 71.9%”

Per-tensor FP8 quantization evaluated

inference quantization “aggregation benchmarks can identify important candidates in FP8, where even per-tensor FP8 precision can be acceptable over 100 tasks at…”

BF16 expert activations evaluated

inference quantization “Expert activations → BF16. Runaway moves by ≈ 0”

Block-scaled NVFP4 KV-cache quantization evaluated

inference quantization “NVFP4 stores 4-bit E2M1 values with one FP8 E4M3 scale per 16-element block”

FP4 quantization of Mamba output-projection weights evaluated

inference quantization “quantizing its weights to FP4 inflates generation length.”

FP8 or NVFP4 KV-cache quantization evaluated

inference quantization “in each case both the key and value caches across attention layers are quantized in FP8 or NVFP4, respectively.”

Increase MoE expert precision and fine-tune on RL rollouts evaluated

inference quantization “Empirically, promoting MoE expert precision and fine-tuning on RL rollouts can mitigate this effect”

KV-cache offloading evaluated

KV cache management “KV-aware routing with session affinity and KV cache offload serves up to 8x as many users”

Linear E2M1 residual correction evaluated

inference quantization “with a linear E2M1 residual the KL falls to 0.00117 against per-tensor FP8’s 0.00134”

MXFP8 block-scaled attention projection quantization evaluated

inference quantization “Substituting MXFP8 for per-tensor FP8 on 𝑞/𝑘/𝑣 changes only the scale granularity with one E8M0 exponent per 32 elements instead of one…”

Promote expert weights from NVFP4 to FP8 evaluated

inference quantization “Routed/shared experts NVFP4→ FP8. In tau3-banking, runaway dropped from 28.2% → 17.9%”

Quantization of the parallel-drafter language-model head evaluated

inference quantization “Further quantizing the LM head improved throughput by another 5%.”

Restricted-search MSE weight-scale selection evaluated

inference quantization “MSE 4-over-6: MSE-based scale selection restricted to a narrower search space.”

Rotation or higher precision for Mamba input activations evaluated

inference quantization “Rotation or higher precision can mitigate this effect in activation”

Sensitivity-driven mixed-precision quantization evaluated

inference quantization “AutoQuant was explored as an alternative sensitivity-driven mixed-precision approach.”

Two-pass FP4 weight quantization with residual correction evaluated

inference quantization “we emulated two-pass FP4 weights, where a second channel corrects the residual left below the amax/24 flush line.”

W4A16 quantization of Mamba output-projection weights evaluated

inference quantization “output verbosity increased by 18.5% at 2.4𝜎, while the runaway rate and knowledge abstention were both unaffected.”

Higher-precision activations for Mamba output projection mentioned

inference quantization “Applying rotation or higher activation precision can mitigate the flush.”

Rotation-based quantization mitigation mentioned

inference quantization “Applying rotation or higher activation precision can mitigate the flush.”

Not used 3

Confidence head not used

decoding strategy “The Lightning 3.5 release used the lightweight Markov correction but omitted this confidence head.”

Mixed-precision key/value cache quantization not used

inference quantization “Per-layer mixed FP8-key/NVFP4-value (or vice-versa) configurations are not evaluated in this comparison.”

W4A4 quantization not used

inference quantization “This tensor analysis stops us from taking the Mamba output projection to W4A4.”

filed at the root 1

NVIDIA NIM used

“NVIDIA NIM packages the model as a production-ready microservice with performance-optimized, validated configurations”

training framework 2

ms-swift used

“ModelScope enables developers to fine-tune the model through its ms-swift framework for specialized multimodal applications.”

Reinforcement-learning environments in NeMo Gym used

“along with the reinforcement learning environments in NeMo Gym.”

infrastructure service 1

NVIDIA Dynamo used

“with the model running on NVIDIA Dynamo, organizations can serve up to 8x more users with 85% lower latency.”

1 more mentioned or evaluated
ModelOpt AutoQuantize evaluated

kernel & quantization library “We use ModelOpt AutoQuantize (AutoQuant) to replace hand-authored layer rules with a sensitivity-constrained mixed-precision search.”

filed at the root 11

Benchmark decontamination core

“The benchmark side (question store, n-gram index, and embedding index) is built once and shared by all runs.”

Chain-of-thought few-shot prompting used

“We evaluate MGSM (Shi et al., 2022) using 8-shot native chain-of-thought prompting in Spanish, German, French, Chinese, Japanese, and…”

Controlled evaluation with fixed settings used

“All models were evaluated under the same evaluation settings including agentic resources (CPU compute and timeouts, tools, search APIs)…”

Candidate-pair overlap gate used

“Judge gate semantic overlap≥0.75 or verbatim overlap≥0.5”

Capability-partition stratification used

“It is important to stratify test cases along with capability partition to assess quantization damage properly.”

Context-length and candidate-count stratification used

“Table 35 stratifies 1,470 MRCR-128k tasks at three repetitions to separate the retrieval from decoding noise.”

Hard same-domain negative examples used

“As negatives we add unmodified questions from a different benchmark in the same domain, chosen to be hard to tell apart from the target”

Overlapping byte-window chunking used

“Chunking 512-byte windows with 256-byte stride (sample and question)”

Rollout non-termination tracking used

“while tracking the failure of a model to terminate as it runs to the context wall”

Separate augmentation and judge models used

“The LLM augmentations are generated with a model different from the judge, so that the judge cannot recognize its own rewrites.”

Synthetic benchmark-question perturbation evaluation used

“transform each with 13 augmentations, at four strengths (0.25, 0.5, 0.75, 0.9)”

benchmark 4

EvalPlus-sanitized benchmark variants used

“using EvalPlus-sanitized variants (Liu et al., 2023).”

Five-shot chain-of-thought prompting for GPQA Diamond used

“GPQA Diamond is evaluated using 5-shot chain-of-thought prompting.”

Three-shot prompting for MBPP-Sanitized used

“MBPP-Sanitized uses 3-shot prompting”

Zero-shot prompting for HumanEval used

“while HumanEval uses zero-shot prompting.”

evaluation harness 3

NeMo Evaluator SDK used

“All evaluation results reported for Nemotron 3.5 Super 120B-A12B Base and Lightning 3.5 30B-A3B Base were collected via NeMo Evaluator SDK”

Benchmark-specific evaluation recipes used

“The published recipes specify the benchmark-specific containers, prompts, inference parameters, parser configurations, and scoring…”

LM Evaluation Harness used

“were collected via NeMo Evaluator SDK and NVIDIA’s open-source container of LM Evaluation Harness”

judge 8

Claude Opus 4.6 judge used

“Judge model Claude Opus 4.6, temperature 0”

JSON-escaped input formatting for judge used

“Passing both texts as JSON-escaped strings keeps their boundaries unambiguous even when the sample itself contains code, markup, or…”

LLM-as-a-Verifier used

“The same checkpoint then serves as a verifier, selecting one candidate using LLM-as-a-Verifier (LAV).”

LLM-judge verification of contamination pairs used

“Each shortlisted pair of a sample and a benchmark question is verified by an LLM judge.”

Order-swapped verifier evaluation used

“For each compared pair and criterion, we evaluate both presentation orders”

Probabilistic Pivot Tournament used

“These pairwise evaluations are organized using LAV’s Probabilistic Pivot Tournament (PPT).”

Teacher-based text-only equivalence judging used

“other free-form answers require same-teacher, text-only equivalence judging with reported confidence≥ 0.90.”

Three-criterion trajectory verification used

“evaluates them against three criteria: specification adherence, output match, and error signal detection.”

averaging - mean@k; avg@k; pass@k; pass^k 3

4-shot sampled pass@1 used

“We additionally evaluate MATH-500 using 4-shot sampled generation with 32 generations per prompt and report pass@1.”

Mean of per-rubric scores used

“We also use the mean of per rubric scores to evaluate responses instead of an overall score.”

Sampled pass@1 used

“We report sampled pass@1 estimated from 32 generations per prompt.”

1 more mentioned or evaluated
Matched supervised fine-tuning checkpoint comparison evaluated

“To measure the effect of mid-training, we ran identical SFT on the base checkpoint and on the mid-trained checkpoint and compared the two.”

filed at the root 1

Postmortem reasoning review used

“We identify the runaway loop by reading reasoning passages in postmortem”

1 more mentioned or evaluated
Ablation of non-attention components evaluated

“We divide and conquer the degradation by ablating non-attention components in pre-MOPD models”

unfiled 22

Hybrid Mamba-Transformer architecture core

“It uses a hybrid Mamba-Transformer architecture”

Interleaved Mamba-2, mixture-of-experts, and attention architecture core

“The model combines interleaved Mamba-2, Mixture-of-Experts (MoE), and attention layers”

Interleaved Mamba-2, mixture-of-experts, and attention hybrid architecture core

“The model combines interleaved Mamba-2, Mixture-of-Experts (MoE), and attention layers”

Joint Transformer processing of the LiLiCorr candidate lattice core

“processes the resulting (B−1)×K candidate lattice jointly with a small Transformer”

V-model iterative unit- and system-level testing and validation used

“Following the V-model methodology, iterative testing and validation at both unit and system levels are essential”

Convert Manipuri translations from Bengali script to Meitei Mayek used

“convert Manipuri translations from Bengali script to Meitei Mayek”

Dual-signal benchmark contamination candidate retrieval used

“The sample is compared with every benchmark question using two signals.”

Embedding-based semantic-overlap detection for benchmark contamination used

“Semantic overlap maximum cosine similarity over chunk pairs”

Embedding-similarity semantic contamination detection used

“Semantic overlap measures the embedding similarity between parts of the two texts.”

Executable visual-tool-use demonstrations for video understanding used

“generate executable SpatialClaw (Cho et al., 2026) demonstrations from existing video question–answer pairs, teaching the student to…”

Left-to-right greedy candidate selection by adjacent-slot compatibility used

“the block is committed from left to right, each slot selecting the candidate most compatible with its committed predecessor”

Measure output length per rollout as a runaway-loop indicator used

“The combination of these two quantities is a useful early-warning indicator.”

Measure rollout truncation rate used

“truncation, the fraction of rollouts containing at least one model call hitting an output limit”

Pre-deployment implementation of guardrails and safety mechanisms used

“Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms…”

Quantizer-enabled language-model GEMM-only QAD updates used

“We optimize only the weights of language-model GEMMs with enabled quantizers. All other parameters remain frozen”

Replay rollout-time MoE routing decisions during policy evaluation used

“We replay the MoE expert-routing decisions recorded during rollout generation when evaluating the policy, reducing train–generation…”

Restore PTQ quantizer configuration and statistics for QAD used

“The quantization configuration and calibrated quantizer statistics are restored from the PTQ checkpoint rather than recalibrated during QAD.”

Restrict QAD optimization to quantized language-model GEMM weights used

“We optimize only the weights of language-model GEMMs with enabled quantizers.”

Select QAD training duration by evaluation rather than assuming more iterations help used

“Overall, our results favor shorter QAD training with 100% SFT data and show that optimal training duration depends on the selected data…”

Speculative decoding, FP8 weights, and FlashAttention for translation efficiency used

“Speculative decoding, FP8 weights, and FlashAttention reduced median translation cost”

Training with open datasets used

“The model ships with open weights, open datasets and training recipes.”

Validate model outputs and use safeguards and human oversight used

“Applications should validate generated answers, code, and tool arguments, and provide appropriate safeguards and human oversight”

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

↑