Model techniques map
ModelsNVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIA · usage rank #5 · released 2026-06-04

NVIDIA-Nemotron-3-Ultra-550B-A55B

What NVIDIA-Nemotron-3-Ultra-550B-A55B’s own documents say it is built from — every method with the sentence that describes it.

OpenRouterHugging Face

Curator’s note550B total / 55B active LatentMoE hybrid Mamba-2 + attention, 1M context, OpenMDW-1.1 (weights + training data + recipes). Best-documented model in this set: a real technical report plus an arXiv mirror.

Rank by open-weight tokens on OpenRouter, week of 2026-09-21.

Against the consensus recipe

Its documents state 10 of the 15 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something the documents do not say, not something the model lacks.

  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 7 of the 19 architecture features checked — 3 of them are not stated in its documents.

Documents say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_nemotron_h.py, configuration_nemotron_h.py) and config.json at revision 77df655d5e, by anthropic/claude-sonnet-5 on 2026-09-25.

FeatureCodeIts documentsEvidence
Grouped-query attention code only in its codenot stated

Full-attention layers use 64 query heads with only 2 key/value heads, repeated via repeat_kv (grouped-query attention).

num_attention_heads = 64num_key_value_heads = 2
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_nemotron_h.py · L847
self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
        self.v_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
modeling_nemotron_h.py · L852–853
Multi-head latent attention not in its codenot stated

The attention module is a standard multi-head/GQA attention with separate q/k/v projections; there is no low-rank KV latent projection or decoupled RoPE key mechanism.

self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
        self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
        self.v_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
modeling_nemotron_h.py · L851–853
Sliding-window attention not in its codenot stated

sliding_window is null in the config and the attention code has no window-masking mechanism applied.

sliding_window = null
Interleaved sliding-window and global attention not in its codenot stated

layers_block_type only distinguishes mamba/moe/attention layers with full causal masks; there is no sliding-window layer type interleaved with global attention.

sliding_window = null
Indexer-selected sparse attention (DeepSeek Sparse Attention) not in its codenot stated

No indexer-based top-k token selection mechanism exists in the code, and no index_topk/index_n_heads config keys are present.

Linear-attention or state-space layers alongside full attention code only in its codenot stated

layers_block_type interleaves mamba (linear_attention) layers with full_attention and moe layers, dispatched via MIXER_TYPES.

layers_block_type = ["mamba", "moe", "mamba", "moe", "mamba", "moe", "mamba", "attention", "moe",…
MIXER_TYPES = {
    "linear_attention": NemotronHMamba2Mixer,
    "full_attention": NemotronHAttention,
    "moe": NemotronHMoE,
    "mlp": NemotronHMLP,
}
modeling_nemotron_h.py · L893–898
Gated DeltaNet layers not in its codenot stated

The recurrent layers implement the Mamba-2 selective SSM (chunked scan with segment sums), not a gated delta-rule mixer like Gated DeltaNet or Kimi Delta Attention.

class NemotronHMamba2Mixer(nn.Module):
modeling_nemotron_h.py · L366
Mamba-2 layers in its code core

Linear-attention layers are implemented as NemotronHMamba2Mixer, a Mamba-2 SSD block with per-head scalar decay (A_log/D), chunked scan, and mamba_num_heads=256, ssm_state_size=128.

mamba_num_heads = 256ssm_state_size = 128
class NemotronHMamba2Mixer(nn.Module):
    """
    Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`.
modeling_nemotron_h.py · L366–368
self.A_log = nn.Parameter(torch.empty(self.num_heads))
modeling_nemotron_h.py · L417
Learnable attention sink not in its codenot stated

No learned per-head sink logit is added to the attention softmax denominator anywhere in the attention code.

Gated attention output not in its codenot stated

The attention module's output goes directly to o_proj with no sigmoid gating multiplication; gating is only used in the Mamba mixer's RMSNormGated, not in softmax attention output.

attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights
modeling_nemotron_h.py · L888–890
QK normalization not in its codenot stated

NemotronHAttention computes q/k/v projections and applies no normalization to query or key states before the dot product.

query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
modeling_nemotron_h.py · L866–868
Partial RoPE code unclearnot stated

partial_rotary_factor is set to 1.0 (full rotary) but the attention forward code shown does not actually call apply_rotary_pos_emb or reference rotary embeddings at all, making the RoPE application path itself unclear from the given files.

partial_rotary_factor = 1.0
YaRN RoPE scaling not in its codenot stated

No rope_scaling key is present in config.json, and no YaRN-specific scaling code path is invoked.

Layers without positional encoding (NoPE) code unclearnot stated

The attention layer's forward pass does not show any RoPE application (no apply_rotary_pos_emb call visible in NemotronHAttention.forward), so it is unclear from the given code whether some full-attention layers omit position encoding while others include it.

query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
modeling_nemotron_h.py · L866–868
Mixture of experts in its code core

MoE layers route tokens to a subset of 512 routed experts via a top-k router selecting 22 experts per token.

n_routed_experts = 512num_experts_per_tok = 22
self.top_k = config.num_experts_per_tok
        self.num_experts = config.num_local_experts
modeling_nemotron_h.py · L730–731
Shared expert in its code used

Each MoE block includes a shared expert MLP (NemotronHMLP with moe_shared_expert_intermediate_size=10240, n_shared_experts=1) applied to every token in addition to routed experts.

n_shared_experts = 1moe_shared_expert_intermediate_size = 10240
self.shared_experts = NemotronHMLP(config=config, intermediate_size=config.moe_shared_expert_intermediate_size)
modeling_nemotron_h.py · L701
hidden_states = hidden_states + self.shared_experts(residuals)
modeling_nemotron_h.py · L723
Auxiliary-loss-free load balancing (selection bias) code only in its codenot stated

The router adds e_score_correction_bias to the sigmoid scores solely to determine top-k expert selection (via topk_group/group masking), while the unbiased scores are gathered for the actual output weights.

scores = router_logits.sigmoid()
        scores_for_choice = scores + self.e_score_correction_bias
modeling_nemotron_h.py · L743–744
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_nemotron_h.py · L759–760
Multi-token prediction layers in its code core

The config declares num_nextn_predict_layers=1 with mtp_layers_block_type set, and the modeling code has weight-skip handling for MTP module keys.

num_nextn_predict_layers = 1mtp_layers_block_type = ["attention", "moe"]
_keys_to_ignore_on_load_unexpected = [r"mtp.*"]
modeling_nemotron_h.py · L981
SwiGLU feed-forward not in its codenot stated

MLP and expert feed-forwards use a single up_proj/down_proj pair with a relu2 activation (act(up(x)) then down_proj), not a SiLU-gated gate/up product.

mlp_hidden_act = "relu2"
self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=config.mlp_bias)
        self.act_fn = ACT2FN[config.mlp_hidden_act]
modeling_nemotron_h.py · L613–615

model architecture 19

token mixer 7

softmax attention 1

Attention layers core

sparse attention 1

fixed-pattern sparse attention 1

Sparse Global Attention Anchors core

linear attention & state space 3

Mamba 3

Mamba-2 core
Mamba core
Mamba-2 SSM cache optional

hybrid layer stacking 2

Hybrid Mamba-Transformer core
Hybrid Mamba-Attention core

channel mixer 8

mixture of experts 8

Mixture of Experts core
Mixture-of-Experts layers core

expert load balancing 2

Expert Parallelism Load Balancing used
MaxVio evaluated

shared experts 1

Shared Experts used

fine-grained experts 1

Granular Mixture of Experts not used

latent mixture of experts 2

LatentMoE core
Hybrid Latent Mixture of Experts core

prediction head 4

Multi-Token Prediction core
Sampled prior-MTP hidden-state conditioning used
Shared-weight design across prediction heads used
Nemotron Hybrid Multi-Token Prediction optional

training objective 7

language modelling objective 1

Sampled-token objective used

multi-token prediction objective 3

Multi-Token Prediction Boosting used
Frozen-backbone MTP-head fine-tuning used
Shared-weight Multi-Token Prediction used

distillation objective 2

Temperature-scaled forward KL distillation used
Logit matching evaluated

auxiliary loss 1

Unfinished-trajectory loss masking used

optimization 29

learning-rate schedule 3

Cosine decay used
Warmup-Stable-Decay used
Learning-rate annealing used

training precision 9

NVFP4 core
NVFP4 pre-training core
BF16 used
MXFP8 used
E2M1 used
FP32 gradient reduction used
Mixed-FP8 quantization used
Two-dimensional block quantization used
BF16 gradient reduction not used

quantization-aware training 2

Stochastic Rounding used
Stochastic Rounding for Mamba Cache used

training stability 1

Weight padding used

training parallelism 5

Expert Parallelism used
Tensor Parallelism used
Pipeline Parallelism used
Context Parallelism used
Data-Weighted Data Parallelism mentioned

training runtime 9

Offline checkpoint merging used
Asynchronous checkpointing used
Asymmetric local-read and single-writer cache paths used
Caching the distributed checkpoint save plan used
In-flight weight updates used
Overlapping NCCL transfers with device-to-host copies used
Persistent checkpoint worker processes used
Persistent shared-storage cache for compiled artifacts used
Seeding node-local storage from a warm shared cache used

data curation 21

filed at the root 1

Nemotron-CC pipeline used

data sourcing 7

GitHub Crawl used
Common Crawl used
EssentialWeb used
FinePDFs used
Nemotron-3-Ultra corpus used
Nemotron-CC used
Nemotron-Post-Training-v3 used

data filtering 5

Heuristic filtering used
Keyword- and regex-based filtering used
Pathological repetition filtering used
Automated verification used
Data filtering pipeline used

deduplication 2

Deduplication used
In-pack deduplication used

synthetic data 3

Knowledge distillation for synthetic data used
Synthetic data generation used
Counterfactual data augmentation mentioned

data mixture & curriculum 2

Gaussian-based data mixture and curriculum construction used
Two-phase curriculum used

sequence packing 1

Length-aware best-fit packing used

post-training 30

supervised fine-tuning 4

Light SFT on teacher-distribution data core
Supervised fine-tuning used
Rejection sampling used
Token-budget-based SFT data blending used

reinforcement learning algorithm 9

Group Relative Policy Optimization used
Reinforcement Learning used
IcePop used
Reinforcement Learning from Verifiable Rewards used
Abstention Training used
Dynamic Abstention-Reward Calibration used
Negative-Advantage Penalties for Malformed Reasoning and Tool Calls used
Pivot Reinforcement Learning used
PPO-Style Policy-Ratio Clipping used

reward modelling 3

Generative Reward Model used
Length-adjusted RL reward used
Principle-conditioned Generative Reward Model used

preference optimization 1

Reinforcement Learning from Human Feedback used

policy distillation 6

Multi-Teacher On-Policy Distillation core
Asynchronous Multi-Teacher On-Policy Distillation used
Behavior–Proximal Policy Decoupling used
IcePop Token-Level Loss Masking used
Multi-Objective Policy Distillation used
Off-Policy Distillation not used

rollout & RL infrastructure 3

One-step off-policy asynchronous reinforcement learning core
Asynchronous reinforcement learning used
On-policy rollouts used

agentic post-training 2

Concurrent multi-environment post-training used
Multi-environment reinforcement learning from verifiable rewards used

mid-training & continual pretraining 2

Continual pretraining used
Continual pretraining for long-context extension used

inference & serving 49

decoding strategy 5

Recursive shared MTP-head drafting core
Speculative decoding default
Multi-Token Prediction used
EAGLE used
Best-of-N scaffolding used

reasoning control 7

Configurable reasoning effort used
Chain-of-thought reasoning used
Generate-verify-refine loop used
Medium-effort reasoning mode used
Turn budget capping used
Turn-aware prompting used
Inference-time reasoning budget control optional

KV cache management 4

Chunked prefill default
Prefix caching used
Periodic cache checkpointing evaluated
8-bit Mamba cache quantization evaluated

inference quantization 20

Mixed-FP8 layers in an NVFP4 recipe core
NVFP4 quantization for routed-expert GEMMs core
Per-tensor FP8 quantization core
Selective retention of BF16 precision core
NVFP4 KV-cache quantization default
FP8 used
FP8 KV-cache quantization used
Post-training quantization used
Four-Over-Six used
SSM cache quantization used
FP16 cache storage with stochastic rounding used
Heuristic mixed per-layer precision quantization used
Random Hadamard transform used
W4A16 quantization used
Block-scaled INT8 quantization with stochastic rounding evaluated
FP8 E4M3 quantization evaluated
Max-based scaling evaluated
MSE-based scaling evaluated
Empirical bits-per-element budget selection evaluated
MSE calibration evaluated

serving parallelism 4

Prefill-decode disaggregation used
Topology-aware NVLink domain placement used
Attention data parallelism optional
Low-precision MoE combine optional

inference scheduling 3

NUMA binding of workers to GPU-local CPU sockets used
Composite-key sorting for topology-aware GPU rank assignment used
Registering NVLink domains as Ray custom resources used

inference kernel 2

MoE-side chunking used
Marlin NVFP4 kernels used

context management 2

Discard-all context management used
Summary-based context compression evaluated

agentic scaffolding 2

Runtime command filter used
Tool-Integrated Reasoning used

software implementation 26

filed at the root 2

Data Designer used
CUDA used

inference engine 4

vLLM used
Environment variable propagation to subprocesses used
SGLang optional
TensorRT-LLM optional

training framework 4

Hugging Face Transformers used
Megatron-LM used
NeMo Gym used
NeMo used

kernel & quantization library 6

FlashInfer used
FlashInfer NVLinkOneSided used
NVIDIA Model-Optimizer used
Container-baked precompiled artifacts used
CUTEDSL optional
CUTLASS optional

agent product 4

OpenCode used
OpenHands used
Droid used
Stirrup used

infrastructure service 6

Dependency unification used
Fresh-clone repository sanitization used
Local container-image caching used
Per-node pooling of initialization actors used
Replacing short-lived Ray actors with tasks used
Ray optional

evaluation 15

benchmark 8

BrowseComp with Search used
KernelBench used
LongBench v2 used
PinchBench used
ProfBench with Search used
ScaleAI Multi Challenge Multi Turn Instruction Following used
SQuAD used
Vals.ai used

evaluation harness 3

Harbor used
NeMo Evaluator SDK used
NeMo Skills used

judge 2

Deterministic tool-call verifier used
LLM-based correctness judging used

human & real-world evaluation 2

Held-out evaluation default
Automated red-teaming loop used

unfiled 12

Transformer core
Amazon Elastic Container Service used
GPT-OSS-120B used
HelpSteer3 used
HotpotQA used
JIT cache management used
NVIDIA API Trial Terms of Service used
Open Model Data Warehouse License Agreement 1.1 used
Single multi-node Slurm launch used
Tau Bench 3 used
Terminus used
three-stage SWE teacher training pipeline used

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated document list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

DocumentKindPublisherStatus
research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Ultra-Technic…technical reportNVIDIAread
arxiv.org/abs/2606.15007technical reportNVIDIAread
research.nvidia.com/labs/nemotron/Nemotron-3-Ultraofficial blogNVIDIA Researchread
nvidianews.nvidia.com/news/nvidia-debuts-nemotron-3-family-of-open-mode…official blogNVIDIA Newsroomread
huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16model cardNVIDIAread
build.nvidia.com/nvidia/nemotron-3-ultra-550b-a55b/modelcardvendor docsNVIDIAread
artificialanalysis.ai/articles/nvidia-nemotron-3-ultra-releasedthird party analysisArtificial Analysisread