Open Models Almanac
ModelsNVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIA · usage rank #11

NVIDIA-Nemotron-3-Ultra-550B-A55B

What NVIDIA-Nemotron-3-Ultra-550B-A55B’s own documents say it is built from — every method with the sentence that describes it.

Parameters
550B / 55B active
Context
1M
Attention
Mamba hybridMamba-2 layers + sparse GQA attention layers
Experts
LatentMoE, 512 routed + 1 shared, top-22
Input
text
Licence
OpenMDW-1.1
Released
2026-06-04

Rank by open-weight tokens on OpenRouter, week of 2026-10-26.

Against the consensus recipe

Its documents state 11 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 7 of the 19 architecture features checked — 3 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_nemotron_h.py, configuration_nemotron_h.py) and config.json at revision 77df655d5e, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention code only in its codenot stated

Full-attention layers use 64 query heads but only 2 key/value heads, with repeat_kv expanding the KV heads to match query groups.

show evidencenum_attention_heads = 64num_key_value_heads = 2
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_nemotron_h.py · L877
    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
    return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
modeling_nemotron_h.py · L839–840
Multi-head latent attention not in its codenot stated

The attention module is a standard multi-head/GQA attention with separate q/k/v projections, no latent KV down-projection or decoupled…

show evidence

The attention module is a standard multi-head/GQA attention with separate q/k/v projections, no latent KV down-projection or decoupled RoPE key as in DeepSeek-V2 MLA.

self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
        self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
        self.v_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
modeling_nemotron_h.py · L881–883
Sliding-window attention not in its codenot stated

sliding_window is null in this config and no windowed masking code is used for the full-attention layers.

show evidencesliding_window = null
Interleaved sliding-window and global attention not in its codenot stated

There is no sliding-window layer type in layers_block_type (only mamba, moe, attention), so no interleaving of local/global attention…

show evidence

There is no sliding-window layer type in layers_block_type (only mamba, moe, attention), so no interleaving of local/global attention occurs.

sliding_window = null
Indexer-selected sparse attention (DeepSeek Sparse Attention) not in its codenot stated

No indexer module or top-k token selection mechanism is present in the attention or mixer code; config has no…

show the whole verdict

No indexer module or top-k token selection mechanism is present in the attention or mixer code; config has no index_topk/index_n_heads/index_head_dim keys.

Linear-attention or state-space layers alongside full attention code only in its codenot stated

layers_block_type mixes mamba (linear_attention) layers with full_attention and moe layers, dispatched via MIXER_TYPES based on block type.

show evidencelayers_block_type = ["mamba", "moe", "mamba", "moe", "mamba", "moe", "mamba", "attention", "moe",…
MIXER_TYPES = {
    "linear_attention": NemotronHMamba2Mixer,
    "full_attention": NemotronHAttention,
    "moe": NemotronHMoE,
    "mlp": NemotronHMLP,
}
modeling_nemotron_h.py · L923–928
Gated DeltaNet layers not in its codenot stated

The recurrent mixer is a Mamba-2 SSM block (selective state-space with conv1d), not a gated delta-rule recurrence; no delta-rule write or…

show the whole verdict

The recurrent mixer is a Mamba-2 SSM block (selective state-space with conv1d), not a gated delta-rule recurrence; no delta-rule write or Kimi Delta Attention code is present.

Mamba-2 layers in its codeadopted

The linear-attention layers are implemented as NemotronHMamba2Mixer, a Mamba-2 selective state-space block with chunked SSD scan and…

show evidence

The linear-attention layers are implemented as NemotronHMamba2Mixer, a Mamba-2 selective state-space block with chunked SSD scan and per-head scalar decay (A_log/D), configured via mamba_num_heads and ssm_state_size.

mamba_num_heads = 256ssm_state_size = 128
class NemotronHMamba2Mixer(nn.Module):
modeling_nemotron_h.py · L389
self.A_log = nn.Parameter(torch.empty(self.num_heads))
modeling_nemotron_h.py · L440
Learnable attention sink not in its codenot stated

No learned per-head sink logit is added to attention weights in eager_attention_forward or elsewhere; the softmax denominator is computed…

show evidence

No learned per-head sink logit is added to attention weights in eager_attention_forward or elsewhere; the softmax denominator is computed over only the key positions.

attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
modeling_nemotron_h.py · L860
Gated attention output not in its codenot stated

NemotronHAttention's output goes directly through o_proj with no sigmoid gate multiplying the attention output; gating (silu) is only used…

show evidence

NemotronHAttention's output goes directly through o_proj with no sigmoid gate multiplying the attention output; gating (silu) is only used inside the Mamba2 mixer's RMSNormGated, not attention.

attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights
modeling_nemotron_h.py · L918–920
QK normalization not in its codenot stated

NemotronHAttention projects q/k/v and applies no RMSNorm/LayerNorm to queries or keys before the dot product.

show evidence
query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
modeling_nemotron_h.py · L896–898
Partial RoPE not in its codenot stated

partial_rotary_factor is 1.0, meaning RoPE is applied to the full head dimension, and no decoupled MLA-style rope key exists.

show evidencepartial_rotary_factor = 1.0
YaRN RoPE scaling code unclearnot stated

No rope_scaling or rope_parameters key is present in the config, and the attention module doesn't reference rotary embeddings or scaling…

show the whole verdict

No rope_scaling or rope_parameters key is present in the config, and the attention module doesn't reference rotary embeddings or scaling at all, so YaRN cannot be confirmed or ruled out from the given files.

Layers without positional encoding (NoPE) code unclearnot stated

NemotronHAttention never calls apply_rotary_pos_emb or references rotary embeddings in its forward, and no rotary embedding module is…

show evidence

NemotronHAttention never calls apply_rotary_pos_emb or references rotary embeddings in its forward, and no rotary embedding module is instantiated in the given files, so whether full_attention layers receive any positional encoding cannot be determined from this code.

query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
modeling_nemotron_h.py · L896–898
Mixture of experts in its codeadopted

MoE blocks route tokens to a subset of 512 routed experts chosen per-token by NemotronHTopkRouter with top-k of 22.

show evidencen_routed_experts = 512num_experts_per_tok = 22
self.top_k = config.num_experts_per_tok
        self.num_experts = config.num_local_experts
modeling_nemotron_h.py · L760–761
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
modeling_nemotron_h.py · L789
Shared expert in its codeadopted

Every MoE layer additionally runs a shared expert MLP of intermediate size 10240 that all tokens pass through, added to the routed-expert…

show evidence

Every MoE layer additionally runs a shared expert MLP of intermediate size 10240 that all tokens pass through, added to the routed-expert output.

n_shared_experts = 1moe_shared_expert_intermediate_size = 10240
self.shared_experts = NemotronHMLP(config=config, intermediate_size=config.moe_shared_expert_intermediate_size)
modeling_nemotron_h.py · L731
hidden_states = hidden_states + self.shared_experts(residuals)
modeling_nemotron_h.py · L753
Auxiliary-loss-free load balancing (selection bias) code only in its codenot stated

The router adds e_score_correction_bias to the sigmoid scores only for top-k expert selection, not for weighting outputs (weights are…

show evidence

The router adds e_score_correction_bias to the sigmoid scores only for top-k expert selection, not for weighting outputs (weights are gathered from the unbiased scores).

scores = router_logits.sigmoid()
        scores_for_choice = scores + self.e_score_correction_bias
modeling_nemotron_h.py · L773–774
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_nemotron_h.py · L789–790
Multi-token prediction layers in its codeadopted

The config declares num_nextn_predict_layers=1 with an mtp_layers_block_type, and the modeling code explicitly ignores mtp.* weights on…

show evidence

The config declares num_nextn_predict_layers=1 with an mtp_layers_block_type, and the modeling code explicitly ignores mtp.* weights on load, indicating MTP modules are declared by the checkpoint.

num_nextn_predict_layers = 1mtp_layers_block_type = ["attention", "moe"]
_keys_to_ignore_on_load_unexpected = [r"mtp.*"]
modeling_nemotron_h.py · L1012
SwiGLU feed-forward not in its codenot stated

Both the dense MLP and the MoE experts use a single up_proj/down_proj pair with a relu2 activation (no gating projection, no SiLU), so…

show evidence

Both the dense MLP and the MoE experts use a single up_proj/down_proj pair with a relu2 activation (no gating projection, no SiLU), so this is not SwiGLU.

mlp_hidden_act = "relu2"
self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=config.mlp_bias)
        self.act_fn = ACT2FN[config.mlp_hidden_act]
modeling_nemotron_h.py · L643–645

token mixer 7

softmax attention 1

Attention layers core

“select Attention layers”

sparse attention 1

fixed-pattern sparse attention 1

Sparse Global Attention Anchors core

“a hybrid Mamba-2 stack with sparse global Attention anchors”

linear attention & state space 3

Mamba 3

Mamba-2 core

“a hybrid Mamba-2 stack with sparse global Attention anchors”

Mamba core

“Nemotron 3 Ultra : Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning”

Mamba-2 SSM cache optional

“--mamba-ssm-cache-dtype float16”

hybrid layer stacking 2

Hybrid Mamba-Transformer core

“Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning”

Hybrid Mamba-Attention core

“employing a Mixture-of-Experts hybrid Mamba-Attention architecture”

channel mixer 5

mixture of experts 5

Mixture of experts core

“Nemotron 3 Ultra : Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning”

expert load balancing 1

Expert Parallelism Load Balancing used

“replicating hot experts across EP ranks (EPLB)”

shared experts 1

Shared expert used

“Shared Expert Intermediate Size 10240”

latent mixture of experts 2

LatentMoE core

“LatentMoE (Elango et al., 2026), which buys more routed experts at fixed inference cost”

Hybrid LatentMoE core

“introduce a breakthrough hybrid latent mixture-of-experts (MoE) architecture”

prediction head 5

Multi-Token Prediction core

“the Ultra model incorporates Multi-Token Prediction (MTP) layers for faster text generation and improved quality”

MTP speculative decoding core

“Multi Token Prediction (MTP) leads to faster inference with speculative decoding”

Shared Multi-Token Prediction head used

“The model includes Multi-Token Prediction (MTP) layers using a shared-weight design across prediction heads”

Hidden-state mixing during recursive MTP training used

“the hidden states passed as input to the MTP at step k are sampled from the set of hidden states produced at MTP steps 1,..,k−1”

Nemotron Hybrid Multi-Token Prediction optional

“--speculative-config '{"method": "nemotron_h_mtp", "num_speculative_tokens": 5}'”

1 more mentioned or evaluated
MaxVio evaluated

expert load balancing “To quantify expert imbalance, we measure the MaxVio metric (DeepSeek-AI, 2025b) which calculates the peak load on any single expert…”

Not used 1

Expert granularity not used

fine-grained experts “LatentMoE helped us achieve better accuracy per parameter than standard Granular MoEs”

language modelling objective 1

Sampled-token objective used

“the sampled-token objective applies supervision only to realized actions”

multi-token prediction objective 3

MTP Boosting used

“The goal of MTP Boosting is to make the MTP head match the backbone’s next-token distribution under the input conditions or noise it…”

Frozen-backbone MTP-head fine-tuning used

“The backbone is fixed for the entire training phase, and only the MTP head receives gradient updates.”

Shared-weight MTP objective used

“As in pre-training, we retain the shared-weight MTP objective during SFT, using two MTP layers with a per-token auxiliary-loss scaling…”

distillation objective 1

Temperature-scaled forward-KL distillation used

“We use a temperature-scaled forward-KL loss against the backbone’s logits.”

auxiliary loss 1

Unfinished-trajectory loss masking used

“We mask the loss on unfinished trajectories”

1 more mentioned or evaluated
Logit matching evaluated

distillation objective “A natural alternative to the sampled-token objective is distribution-level distillation, where the student is trained to match the…”

learning-rate schedule 3

Cosine decay used

“using a cosine learning-rate schedule with peak learning rate 1.5×10−5”

Warmup-Stable-Decay used

“Warmup-Stable-Decay (WSD) learning rate schedule over a total horizon of 20 trillion tokens”

Learning-rate annealing used

“starting learning rate annealing (both a 5T and 10T decay) immediately after rolling back to the 15T token checkpoint mitigates divergence”

training precision 8

NVFP4 core

“The majority of linear layers use NVFP4 for weights, activations, and gradients”

NVFP4 pretraining core

“it is trained using an NVFP4 pre-training recipe to maximize compute efficiency”

BF16 precision used

“BF16 precision for attention linear layers, MoE latent projection layers”

E2M1 floating-point format used

“NVFP4 layers use the E2M1 datatype”

FP32 gradient reduction used

“Rolling back to an earlier checkpoint and moving back to the full FP32 gradient reduction recipe re-stabilized training”

FP8 quantization of selected layers used

“this step is precisely the introduction of the mixed-FP8 layers on top of the NVFP4-amax recipe”

MXFP8 used

“select layers (including latent projections, MTP layers, QKV/attention projections, and embeddings) are maintained in BF16 or MXFP8 for…”

Two-dimensional block quantization used

“NVFP4 layers use the E2M1 datatype with two-dimensional block quantization on weights”

quantization-aware training 1

Stochastic Rounding used

“FP16 with stochastic rounding preserves FP32-cache accuracy”

training stability 1

Weight padding used

“the route we have taken is to pad the affected weight matrices at load time”

training parallelism 4

Expert parallelism used

“wide EP the practical choice for high-throughput serving”

Tensor parallelism used

“wide TP the practical choice for low-latency serving”

Context parallelism used

“32-way context parallelism”

Pipeline parallelism used

“2-way pipeline parallelism”

training runtime 9

Checkpoint merging used

“offline checkpoint merging for evaluation analysis”

Asynchronous checkpointing used

“Enabling asynchronous checkpointing from the Nvidia Resiliency Extension (NVRx), where model parameters are copied to CPU and persisted in…”

Asymmetric local-read and single-writer cache paths used

“At job completion, a single designated sidecar process archives the caches back to shared storage as compressed tarballs, rather than all…”

Caching the distributed checkpoint save plan used

“we cache the distributed save plan so it is computed once rather than on every save.”

In-flight weight updates used

“leveraging in-flight weight updates and MTP to accelerate rollout generation”

Overlapping NCCL transfers with device-to-host copies used

“overlapped NCCL transfers with D2H copies to shrink the synchronous staging window”

Persistent checkpoint worker processes used

“persistent checkpoint worker processes that avoid fork/spawn overhead on each save”

Persistent shared-storage cache for compiled artifacts used

“Persistent warm cache on shared storage.At job completion, all JIT artifacts ... are written back to a persistent shared directory as…”

Seeding node-local storage from a warm shared cache used

“Before Ray initialization, a setup script on each node extracts the warm cache tarballs into node-local/tmp”

1 more mentioned or evaluated
Data-Weighted Data Parallelism mentioned

training parallelism “DWDP (nvidia2026dwdp) keeps execution data-parallel and instead pulls each layer's expert weights to every rank”

Not used 1

BF16 gradient reduction not used

training precision “The first divergence, which occurred at around 8T tokens, was attributed to a reduction in local gradient accumulation precision for the…”

filed at the root 1

Nemotron-CC pipeline used

“The data was subsequently deduplicated and filtered in various ways described in the Nemotron-CC paper”

data sourcing 7

GitHub Crawl used

“The GitHub Crawl was collected using the GitHub REST API and the Amazon S3 API”

Common Crawl used

“The English Common Crawl data was downloaded from the Common Crawl Foundation”

EssentialWeb used

“These datasets, such as FinePDFs, EssentialWeb, HotpotQA, SQuAD, and HelpSteer3”

FinePDFs used

“These datasets, such as FinePDFs, EssentialWeb, HotpotQA, SQuAD, and HelpSteer3”

Nemotron-3-Ultra corpus used

“The foundation of the model is trained on the Nemotron-3-Ultra corpus”

Nemotron-CC used

“Nemotron-CC paper”

Nemotron-Post-Training-v3 used

“Major portions of the fine-tuning corpus are released in the Nemotron-Post-Training-v3 collection”

data filtering 5

Heuristic filtering used

“we applied just heuristic filtering instead—similar to what we did for lower quality English data in the Nemotron-CC pipeline”

Keyword- and regex-based content filtering used

“we apply targeted keyword- and regex-based filters and remove all trajectories matching such behavior”

N-gram repetition filtering used

“We then aggressively filter reasoning traces exhibiting pathological repetition, such as repeated n-grams within a sliding window or…”

Automated verification used

“Across all pipelines we stack automated verification—compilers, numerical checks, language identification—to ensure our data is high quality”

Data filtering pipeline used

“we apply a unified data filtering pipeline to ensure that only high-quality, license-compliant, and verifiable samples are used for…”

deduplication 2

Corpus deduplication used

“The data was subsequently deduplicated and filtered in various ways”

In-pack deduplication used

“we enforce an in-pack deduplication constraint to prevent identical prompts from co-occurring within the same sequence.”

synthetic data 2

Knowledge distillation for synthetic data generation used

“we generate synthetic data by distilling trajectories, solutions, and translations from strong teacher models”

Synthetic data generation used

“During post-training, we generate synthetic data by distilling trajectories, solutions, and translations from strong teacher models”

data mixture & curriculum 2

Gaussian-based data-mixture and curriculum construction used

“For data mixture and curriculum construction, we adopt the Gaussian-based approach introduced in NVIDIA (2025b).”

Two-phase curriculum used

“two-phase curriculum and transition from a data mixture which biases dataset diversity ... to data mixture which biases dataset quality”

sequence packing 1

Length-aware best-fit packing used

“we adopt a length-aware best-fit packing strategy (Ding et al., 2024), which packs multiple conversations into sequences up to a maximum…”

1 more mentioned or evaluated
Counterfactual data augmentation mentioned

synthetic data “mitigation strategies like counterfactual data augmentation to align with the desired model behavior”

supervised fine-tuning 4

Light supervised fine-tuning on teacher-distribution data core

“the student undergoes a very light SFT on data drawn from the teacher’s training distribution.”

Supervised fine-tuning used

“To train the student model, we start from the pre-trained base model and perform Supervised Fine-Tuning (SFT) in two stages”

Rejection sampling used

“rejection sampling on those reasoning traces”

Token-budget-based SFT data blending used

“We construct the final SFT mixture using token-level target proportions rather than example counts”

reinforcement learning algorithm 9

Reinforcement learning used

“post-trained using Supervised Fine Tuning (SFT), Reinforcement Learning (RL), and Multi-teacher On-Policy Distillation”

Group Relative Policy Optimization used

“Our training procedure largely follows the asynchronous GRPO algorithm with the stability optimizations proposed in NVIDIA (2026)”

Reinforcement learning from verifiable rewards used

“we conduct a unified RLVR (Reinforcement Learning with Verifiable Reward) training stage spanning all available environments”

Abstention training used

“the teacher also underwent abstention training, where the model was encouraged to abstain when uncertain rather than hallucinate incorrect…”

IcePop used

“where represents token-level masking using IcePop strategy (ring1t)”

Dynamic abstention-reward calibration used

“We dynamically calibrated the abstention reward throughout training”

Negative-advantage penalties for malformed reasoning and tool calls used

“we also penalize malformed reasoning and tool calls by assigning negative advantage to the offending tokens.”

PivotRL used

“proceeded with pivot RL in the MOPD stage, distilling the SFT-trained teacher into the student Ultra model”

PPO-style policy-ratio clipping used

“PPO-style clipping is applied to r_t(θ) around the proximal policy πprox.”

reward modelling 3

Generative Reward Model used

“Generative Reward Models (GenRM) (Wang2025HelpSteer3Preference) with reasoning capabilities help mitigate such reward hacking behaviors”

Length penalty used

“length-based adjustments are applied to the RL rewards for them.”

Principle-conditioned generative reward model used

“A principle-following GenRM makes this process much more flexible”

preference optimization 1

Reinforcement Learning from Human Feedback (RLHF) used

“Conversational quality was further refined through RLHF”

policy distillation 4

Multi-Teacher On-Policy Distillation core

“MOPD provides a dense token-level learning signal from the relevant teacher distribution.”

Asynchronous Multi-Teacher On-Policy Distillation used

“In our implementation, MOPD is executed asynchronously.”

Behavior–Proximal Policy Decoupling used

“To stabilize this setting, we decouple the behavior policy from the proximal policy”

IcePop Token-Level Loss Masking used

“m_t represents token-level masking using IcePop strategy”

rollout & RL infrastructure 3

One-step off-policy asynchronous reinforcement learning core

“During RL and MOPD, we train using a one-step off-policy asynchronous RL setup, so rollout generation is overlapped with the policy update”

Asynchronous reinforcement learning used

“It utilized an asynchronous RL architecture that fully decouples training from inference across separate GPU devices”

On-policy rollout generation used

“This technique uses strong teacher models to guide training on the model's own generated attempts (on-policy rollouts)”

agentic post-training 2

Multi-environment reinforcement learning used

“unified RLVR over a wide mix of reasoning, agentic, code, safety, usability, and chat environments”

Concurrent multi-environment post-training used

“advanced reinforcement learning techniques with concurrent multi-environment post-training at scale”

mid-training & continual pretraining 2

Long-context extension via continued pretraining used

“added a long-context phase (LC-Phase) at the end of pretraining”

Continued pretraining used

“continuous pretraining (CPT) to equip the base model with long-context ability”

Not used 1

Off-Policy Distillation not used

policy distillation “yielding stronger gains than purely off-policy distillation”

decoding strategy 5

Recursive shared MTP-head drafting core

“At each decoding iteration, the MTP head is applied recurrently to propose 𝑘 candidate tokens, which the base model verifies in a single…”

Speculative decoding default

“enables faster inference via native speculative decoding”

Multi-Token Prediction used

“In order to accelerate rollout generation, we use speculative decoding with Multi-Token Prediction (MTP).”

EAGLE used

“speculative-algorithm EAGLE”

Best-of-N selection used

“best-of-n selection traces”

reasoning control 6

Thinking mode used

“Ultra is also equipped with reasoning effort control, which supports inference-time adjustment of the accuracy–compute trade-off”

Chain-of-thought reasoning used

“generate both COT and TIR solution trajectories with DeepSeek-V4-Pro”

Generate-verify-refine used

“Following the generate–verify–refine methodology of Shao et al. (2025) and Nemotron-Cascade-2 (Yang et al., 2026)”

Turn budget capping used

“Each model runs Terminal-Bench v2.1 at four budgets - 10, 20, 50 and 100 turns”

Turn-aware prompting used

“we customized the Terminus 2 harness so that every run states the turn budget up front... and warns the model as it approaches the limit”

Reasoning token-budget control optional

“The regular and medium-effort reasoning modes have the option to be used in conjunction with inference-time budget control.”

KV cache management 2

Chunked prefill default

“All deployment snippets below default to port 8000, with chunked prefill, NVFP4 KV caching, and MTP (5 speculative tokens) enabled”

Prefix caching used

“enable-prefix-caching”

inference quantization 14

Per-tensor FP8 quantization core

“FP8 per-tensor GEMMs for shared experts and Mamba linear layers; static max-calibrated per-tensor scales.”

Mixed-FP8 layers in an NVFP4 recipe core

“We therefore selected5.03 BPE(NVFP4 with mixed-FP8) as the operating point”

NVFP4 routed-expert GEMM quantization core

“NVFP4 routed-expert GEMMs; Dynamic max-based activation scaling and max-calibrated 4/6 weight scaling.”

Selective BF16 retention for sensitive layers core

“BF16 precision for attention linear layers, MoE latent projection layers (i.e, layers for which accuracy degradation outweighed the…”

NVFP4 KV-cache quantization default

“All deployment snippets below default to port 8000, with chunked prefill, NVFP4 KV caching, and MTP (5 speculative tokens) enabled”

FP8 used

“KV Cache is quantized to FP8”

FP8 KV-cache quantization used

“KV Cache is quantized to FP8”

Post-training quantization used

“We apply post-training quantization (PTQ) using Model-Optimizer to quantize the Nemotron 3 Ultra checkpoint to NVFP4”

Four-Over-Six used

“Four-Over-Six selects between the 𝑀 = 4 and 𝑀 = 6 weight grids per block, using the option that minimizes reconstruction error.”

Mamba SSM-cache quantization used

“we quantize the Mamba SSM cache from the original FP32 precision to lower precisions”

FP16 cache storage with stochastic rounding used

“we use FP16 SSM cache storage with stochastic rounding”

Heuristic mixed per-layer precision quantization used

“We begin from a heuristic mixed per-layer precision recipe informed by Model-Optimizer AutoQuantize sensitivity analysis”

Random Hadamard transform used

“Random Hadamard Transforms on inputs to wgrad”

W4A16 quantization used

“W4A16 (weights NVFP4, activations BF16)”

serving parallelism 4

Prefill-decode disaggregation used

“Prefill-decode disaggregation, which runs the two phases on separate workers with their own parallelism and scheduling, is the established…”

Topology-aware NVLink domain placement used

“The fix was to make rank assignment domain-aware and ensure all GPUs within an EP group are co-located on the same rack.”

Attention data parallelism optional

“enable_attention_dp: true”

Low-precision MoE combine optional

“use_low_precision_moe_combine: true”

inference scheduling 3

NUMA binding for GPU-local CPU affinity used

“The fix was to explicitly bind policy and vLLM worker processes to the CPU socket local to their assigned GPUs.”

Composite-key topology-aware GPU rank assignment used

“bundle indices are sorted by a( domain_min_topo_rank,topo_rank,gpu_id )composite key”

NVLink domains as Ray custom resources used

“and registers it as a Ray custom resource (nvlink_domain_<ClusterUUID>), giving the scheduler rack-membership information it previously…”

inference kernel 2

MoE-side chunking used

“The fix is MoE-side chunking, splitting the token batch within the MoE kernel itself”

Marlin NVFP4 kernels used

“Marlin NVFP4 linear and MoE kernels on Hopper that impose tile- and thread-level alignment constraints”

context management 1

Discard-all context management used

“We focus primarily on discard-all context management, where earlier search observations are removed once the interaction history exceeds…”

agentic scaffolding 2

Runtime command filtering used

“we install a runtime command filter that blocks every way the agent could pull that history back over the network”

Tool-Integrated Reasoning used

“generate both COT and TIR solution trajectories”

9 more mentioned or evaluated
MSE-based weight-scale selection evaluated

inference quantization “MSE calibration gave an additional weight-MSE reduction of 27.1%, downstream evaluations did not show a consistent accuracy increase”

Periodic cache checkpointing evaluated

KV cache management “We ran initial experiments on Nemotron 3 Super NVFP4, using emulated quantization”

Block-scaled INT8 quantization with stochastic rounding evaluated

inference quantization “Block-scaled INT8 quantization with stochastic rounding largely preserves FP32-cache accuracy”

FP8 E4M3 quantization evaluated

inference quantization “FP8 E4M3 quantization degrades accuracy”

Max-based weight scaling evaluated

inference quantization “we experimented with max-based, MSE-based, and Four-Over-Six scaling”

8-bit Mamba cache quantization evaluated

KV cache management “we also explore 8-bit Mamba cache quantization formats. We evaluated this on Nemotron 3 Super.”

Empirical bits-per-element budget selection evaluated

inference quantization “We selected the model’s bits-per-element (BPE) budget empirically, by quantizing a fixed intermediate checkpoint at a range of BPE…”

Summary-based context compression evaluated

context management “including discard-all resets and summary-based compression.”

effort parameters mentioned

reasoning control “There are a range of approaches to this today including model ‘effort’ parameters, token limits at the request or task level, or turn…”

filed at the root 2

Data Designer used

“Data Designer is one of the libraries used to prepare these corpora”

CUDA used

“software frameworks (e.g., CUDA libraries)”

inference engine 4

vLLM used

“disaggregation for hybrid Mamba-Attention models now works out-of-the-box in vLLM (Kwon et al., 2023)”

Environment-variable propagation to subprocesses used

“Library paths and environment variables were explicitly forwarded to all subprocesses at spawn time.”

SGLang optional

“SGLang deployment snippet provided”

TensorRT-LLM optional

“VLLM_FLASHINFER_MOE_BACKEND=latency(TRTLLM-Gen)”

training framework 4

Hugging Face Transformers used

“We used transformers on a single node with CPU offloading to quantize Ultra layer-by-layer.”

Megatron-LM used

“we chose Megatron-LM for its efficient multi-node distributed parallelism and MoE support”

NeMo Gym used

“The RL environments and datasets are released as part of NeMo Gym”

NeMo used

“Runtime Engine(s): NeMo 26.04.01”

kernel & quantization library 6

FlashInfer used

“mamba-backend flashinfer”

FlashInfer NVLinkOneSided all-to-all used

“adopted FlashInfer’s NVLinkOneSided implementation... as the best end-to-end choice for Nemotron 3 Ultra serving on GB200 NVL72”

NVIDIA Model-Optimizer used

“Model-Optimizer supports both HuggingFace and Megatron-LM for quantization”

Container-baked precompiled artifacts used

“precompiled cubins were baked into the container image at build time viaflashinfer download-cubin”

CUTEDSL optional

“moe_config: backend: CUTEDSL”

CUTLASS optional

“VLLM_FLASHINFER_MOE_BACKEND=throughput(CUTLASS)”

agent product 4

OpenCode used

“the model is trained under at least two of the following harnesses: • Stirrup • OpenHands • OpenCode • Terminus • Droid”

OpenHands used

“the model is trained under at least two of the following harnesses: • Stirrup • OpenHands • OpenCode • Terminus • Droid”

Droid used

“the model is trained under at least two of the following harnesses: • Stirrup • OpenHands • OpenCode • Terminus • Droid”

Stirrup used

“the model is trained under at least two of the following harnesses: • Stirrup • OpenHands • OpenCode • Terminus • Droid”

infrastructure service 6

Dependency version unification used

“All components of the RL stack were aligned to a single version of shared GPU kernel libraries, ensuring ABI consistency across the job.”

Fresh-clone repository sanitization used

“we rewrite the in-container repository to look like a fresh clone taken at the moment of the task’s base commit”

Local container-image caching used

“Enroot’s local squashfs cache ensures that once a node has extracted the container image, subsequent jobs reuse the cached local copy”

Per-node pooling of initialization actors used

“We eliminated 40% of actor registrations by converting short-lived actors to tasks, pooling initialization actors per node, and applied…”

Replacing short-lived Ray actors with tasks used

“We eliminated 40% of actor registrations by converting short-lived actors to tasks, pooling initialization actors per node, and applied…”

Ray optional

“The recommended multi-processing backend for multi-node deployments is Ray v2”

benchmark 8

BrowseComp with Search used

“BrowseComp with Search, Tau Bench 3, ProfBench with Search, PinchBench, Vals.ai, LongBench v2”

KernelBench used

“the evaluations also used dedicated open-source packaged containers for ScaleAI Multi Challenge Multi Turn Instruction Following and…”

LongBench v2 used

“BrowseComp with Search, Tau Bench 3, ProfBench with Search, PinchBench, Vals.ai, LongBench v2”

PinchBench used

“BrowseComp with Search, Tau Bench 3, ProfBench with Search, PinchBench, Vals.ai, LongBench v2”

ProfBench with Search used

“BrowseComp with Search, Tau Bench 3, ProfBench with Search, PinchBench, Vals.ai, LongBench v2”

ScaleAI Multi Challenge Multi Turn Instruction Following used

“the evaluations also used dedicated open-source packaged containers for ScaleAI Multi Challenge Multi Turn Instruction Following and…”

SQuAD used

“These datasets, such as FinePDFs, EssentialWeb, HotpotQA, SQuAD, and HelpSteer3”

Vals.ai used

“BrowseComp with Search, Tau Bench 3, ProfBench with Search, PinchBench, Vals.ai, LongBench v2”

evaluation harness 3

NeMo Evaluator SDK used

“All evaluation results were collected via Nemo Evaluator SDK”

Harbor Framework used

“We used three main evaluation harnesses: Nemo Gym, Nemo Skills, and Harbor”

NeMo Skills used

“We used three main evaluation harnesses: Nemo Gym, Nemo Skills, and Harbor”

judge 2

Deterministic tool-call verifier used

“a deterministic verifier marks an injection as resisted only if the agent does not invoke the attacker’s target tool with the target…”

LLM-based correctness judging used

“We then check each trajectory against the reference answer using an LLM-based judging pipeline”

human & real-world evaluation 2

Held-out evaluation benchmarks default

“These benchmarks were reserved for final validation: they were not used for training-time monitoring, checkpoint selection, or any other…”

Automated red-teaming for indirect prompt injection used

“we use an automated red-teaming loop in which an attacker model iteratively rewrites the injected instruction against a defender model”

unfiled 12

Transformer core

“Nemotron 3 Ultra : Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning”

Amazon Elastic Container Service used

“extended sandboxing support via AWS ECS on Nemo Evaluator”

GPT-OSS-120B used

“use open source permissive models such as GPT-OSS-120B to produce step-by-step reasoning traces”

HelpSteer3 used

“These datasets, such as FinePDFs, EssentialWeb, HotpotQA, SQuAD, and HelpSteer3”

HotpotQA used

“These datasets, such as FinePDFs, EssentialWeb, HotpotQA, SQuAD, and HelpSteer3”

JIT cache management used

“The fix was a three-pronged cache management strategy”

NVIDIA API Trial Terms of Service used

“The trial service is governed by the NVIDIA API Trial Terms of Service”

Open Model Data Warehouse License Agreement 1.1 used

“Use of this model is governed by the OpenMDW License Agreement, version 1.1 (OpenMDW-1.1)”

Single multi-node Slurm launch used

“Restructuring to a single multi-node srun reduced Slurm controller interactions to 𝑂(1).”

Tau Bench 3 used

“BrowseComp with Search, Tau Bench 3, ProfBench with Search, PinchBench, Vals.ai, LongBench v2”

Terminus used

“the model is trained under at least two of the following harnesses: • Stirrup • OpenHands • OpenCode • Terminus • Droid”

three-stage SWE teacher training pipeline used

“The SWE teacher was trained through a three-stage pipeline.”

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

↑