Model techniques map
ModelsNVIDIA-Nemotron-3.5-Lightning-30B-A3B

NVIDIA · released 2026-08-11

NVIDIA-Nemotron-3.5-Lightning-30B-A3B

What NVIDIA-Nemotron-3.5-Lightning-30B-A3B’s own documents say it is built from — every method with the sentence that describes it.

OpenRouterHugging Face

Curator’s note30B total / 3B active hybrid: interleaved Mamba-2 and MoE layers with select attention layers (architecture id nemotron_h), >20T pre-training tokens, up to 1M context, OpenMDW-1.1 (weights + data + recipes). Multi-Token Prediction is trained in during pre-training, and the release ships DSpark and DFlash draft models plus an NVFP4 checkpoint alongside BF16. The curated URL is the BF16 reference release — the customization starting point. No Nemotron 3.5 report yet; arXiv 2512.20856 is the Nemotron 3 family report.

Against the consensus recipe

Its documents state 6 of the 15 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something the documents do not say, not something the model lacks.

  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 7 of the 19 architecture features checked — 4 of them are not stated in its documents.

Documents say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_nemotron_h.py, configuration_nemotron_h.py) and config.json at revision a9904d24bc, by anthropic/claude-sonnet-5 on 2026-09-25.

FeatureCodeIts documentsEvidence
Grouped-query attention code only in its codenot stated

Full-attention layers project 32 query heads down to 2 KV heads which are repeated via repeat_kv before the dot product.

num_attention_heads = 32num_key_value_heads = 2
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_nemotron_h.py · L847
self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
        self.v_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
modeling_nemotron_h.py · L852–853
Multi-head latent attention not in its codenot stated

The attention module uses standard per-head q/k/v projections with no low-rank KV latent or decoupled RoPE key mechanism.

self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
        self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
        self.v_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
modeling_nemotron_h.py · L851–853
Sliding-window attention not in its codenot stated

sliding_window is null in this config and no windowed mask is constructed for full_attention layers.

sliding_window = null
Interleaved sliding-window and global attention not in its codenot stated

layers_block_type interleaves mamba, moe, and full_attention layers, but none are sliding-window since sliding_window is null.

sliding_window = nulllayers_block_type = ["mamba", "moe", "mamba", "moe", "mamba", "attention", "moe", "mamba", "moe",…
Indexer-selected sparse attention (DeepSeek Sparse Attention) not in its codenot stated

No indexer-based top-k token selection mechanism appears anywhere in the code or config keys.

Linear-attention or state-space layers alongside full attention code only in its codenot stated

layers_block_type mixes NemotronHMamba2Mixer linear-attention layers with NemotronHAttention full_attention layers and moe/mlp layers, dispatched through MIXER_TYPES.

layers_block_type = ["mamba", "moe", "mamba", "moe", "mamba", "attention", "moe", "mamba", "moe",…
MIXER_TYPES = {
    "linear_attention": NemotronHMamba2Mixer,
    "full_attention": NemotronHAttention,
    "moe": NemotronHMoE,
    "mlp": NemotronHMLP,
}
modeling_nemotron_h.py · L893–898
Gated DeltaNet layers not in its codenot stated

The linear-attention layers implement Mamba-2 style SSM scan (segment sum, chunked SSD), not a gated delta rule / Kimi Delta Attention recurrence.

Mamba-2 layers in its code core

NemotronHMamba2Mixer implements the Mamba-2 chunked SSD scan with per-head scalar decay (A_log/D), used for layer type 'mamba' (linear_attention) per layers_block_type.

mamba_num_heads = 64ssm_state_size = 128
class NemotronHMamba2Mixer(nn.Module):
modeling_nemotron_h.py · L366
self.A_log = nn.Parameter(torch.empty(self.num_heads))
modeling_nemotron_h.py · L417
Learnable attention sink not in its codenot stated

No per-head learned sink logit is added to the attention softmax denominator anywhere in the attention implementation.

Gated attention output not in its codenot stated

NemotronHAttention's output goes straight from attn_output to o_proj with no sigmoid gate multiplication; gating only appears in the Mamba mixer's RMSNormGated, which is not attention output gating.

attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights
modeling_nemotron_h.py · L888–890
QK normalization not in its codenot stated

NemotronHAttention computes q/k/v projections and applies attention directly with no RMSNorm/LayerNorm applied to queries or keys.

query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
modeling_nemotron_h.py · L866–868
Partial RoPE not in its codenot stated

partial_rotary_factor is 1.0, and no rotary embedding application is even visible in NemotronHAttention.forward (no cos/sin/apply_rotary_pos_emb call), so full or no RoPE is applied uniformly with no partial split.

partial_rotary_factor = 1.0
YaRN RoPE scaling code unclearnot stated

The config has no rope_scaling key at all, and the attention forward pass shown does not call apply_rotary_pos_emb or reference rope_scaling, so YaRN cannot be confirmed or ruled out from these files.

Layers without positional encoding (NoPE) code unclearnot stated

No rotary embedding application (cos/sin) is visible inside NemotronHAttention.forward in this file, so it's unclear whether full_attention layers receive positional encoding uniformly or whether some lack it; the rotary embedding module itself is not included here.

Mixture of experts in its code core

NemotronHTopkRouter routes each token to num_experts_per_tok=6 of n_routed_experts=128 expert MLPs via sigmoid scores and top-k selection.

n_routed_experts = 128num_experts_per_tok = 6
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_nemotron_h.py · L759–760
Shared expert code only in its codenot stated

NemotronHMoE adds a shared_experts NemotronHMLP with moe_shared_expert_intermediate_size=3712 whose output is added to every token unconditionally, and n_shared_experts=1.

n_shared_experts = 1moe_shared_expert_intermediate_size = 3712
self.shared_experts = NemotronHMLP(config=config, intermediate_size=config.moe_shared_expert_intermediate_size)
modeling_nemotron_h.py · L701
hidden_states = hidden_states + self.shared_experts(residuals)
modeling_nemotron_h.py · L723
Auxiliary-loss-free load balancing (selection bias) code only in its codenot stated

The router adds e_score_correction_bias to sigmoid scores only for selecting top-k experts, while output weighting uses the unbiased scores.

scores = router_logits.sigmoid()
        scores_for_choice = scores + self.e_score_correction_bias
modeling_nemotron_h.py · L743–744
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_nemotron_h.py · L759–760
Multi-token prediction layers in its code core

config declares num_nextn_predict_layers=1 and mtp_layers_block_type ['attention','moe'], enabling multi-token prediction module construction.

num_nextn_predict_layers = 1mtp_layers_block_type = ["attention", "moe"]
SwiGLU feed-forward not in its codenot stated

Both the dense NemotronHMLP and the MoE experts use a single up_proj followed by a relu2 activation and down_proj with no gating multiplication, i.e. not SwiGLU.

mlp_hidden_act = "relu2"
self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=config.mlp_bias)
        self.act_fn = ACT2FN[config.mlp_hidden_act]
modeling_nemotron_h.py · L613–615

model architecture 10

token mixer 3

linear attention & state space 1

Mamba 1

Mamba-2 core

channel mixer 5

mixture of experts 5

Mixture of Experts core
MoE with 128 routed experts and a shared expert used

expert routing 1

Token-level expert routing core

latent mixture of experts 2

LatentMoE core
Latent-dimension reduction used

positional encoding 1

Omitting RoPE in attention layers core

prediction head 1

Multi-Token Prediction core

training objective 3

multi-token prediction objective 3

Multi-Token Prediction used
Multi-Token Prediction Boosting used
Continued pretraining for MTP layers used

optimization 9

training precision 6

NVFP4 core
BF16 used
NVFP4 pre-training used
MXFP8 used
High-precision final network layers used
NVFP4 fine-grained micro-block scaling used

quantization-aware training 3

NVFP4 Training used
Random Hadamard Transforms used
Stochastic Rounding of Gradients used

data curation 12

data sourcing 1

GitHub Crawl used

data filtering 6

Heuristic filtering used
Keyword- and regex-based filtering used
Pathological repetition filtering used
Unified data filtering pipeline used
Permissive-license filtering used
Structural checks for malformed examples used

deduplication 1

Deduplication used

synthetic data 4

Knowledge distillation for synthetic data used
Synthetic data generation, filtering, and curation used
Synthetic privacy-preserving personas used
Counterfactual data augmentation mentioned

post-training 11

supervised fine-tuning 4

Supervised fine-tuning used
LoRA fine-tuning used
Supervised fine-tuning at 256k sequence length used
Fine-tuning with demographically balanced datasets mentioned

reinforcement learning algorithm 2

Group Relative Policy Optimization used
GRPO with Masked Importance Sampling used

reward modelling 1

Reinforcement learning with verifiable rewards used

rollout & RL infrastructure 1

Asynchronous reinforcement learning used

agentic post-training 2

Multi-environment reinforcement learning core
Harness-optimized training used

mid-training & continual pretraining 1

Continual pretraining for long-context extension used

inference & serving 11

decoding strategy 6

Speculative decoding core
Multi-Token Prediction used
Best-of-N scaffolding used
DSpark optional
DFlash optional
Concurrency-aware draft-length tuning evaluated

reasoning control 1

Inference-time reasoning budget control used

inference quantization 2

Post-training quantization used
NVFP4 quantization optional

agentic scaffolding 2

Frontier-planning to execution-model routing core
Bash computer-use agent used

software implementation 5

filed at the root 2

Nemotron Parse used
Model routing and orchestration optional

inference engine 1

TensorRT-LLM used

infrastructure service 2

NeMo Guardrails used
NeMo Switchyard used

evaluation 4

filed at the root 1

V-model iterative testing and validation mentioned

benchmark 2

Verified CUDA kernels used
Artificial Analysis Intelligence Index evaluated

evaluation harness 1

NeMo Gym and NeMo Evaluator-based harness used

unfiled 8

Model specialization for agent execution core
Configurable routing profiles for model selection used
frontier/execution model division of labor used
OpenAI-compatible API endpoint used
Reinforcement learning datasets with trajectories, tool calls, and preference signals used
Reproducible evaluation recipes in NeMo Gym used
task-to-best-model routing with NeMo Switchyard optional
Bias audits mentioned

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated document list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

DocumentKindPublisherStatus
developer.nvidia.com/blog/nvidia-nemotron-3-5-lightning-delivers-fast-a…official blogNVIDIA Technical Blogread
arxiv.org/abs/2512.20856technical reportNVIDIAread
huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16model cardNVIDIAread
build.nvidia.com/nvidia/nemotron-3.5-lightning-30b-a3b/modelcardvendor docsNVIDIAread
developer.nvidia.com/nemotronvendor docsNVIDIAread
openrouter.ai/nvidia/nemotron-3.5-lightningvendor docsOpenRouterread
analyticsvidhya.com/blog/2026/08/nvidia-nemotron-3-5-lightningthird party analysisAnalytics Vidhyaread