Open Models Almanac
ModelsNVIDIA-Nemotron-3.5-Lightning-30B-A3B

NVIDIA

NVIDIA-Nemotron-3.5-Lightning-30B-A3B

What NVIDIA-Nemotron-3.5-Lightning-30B-A3B’s own documents say it is built from — every method with the sentence that describes it.

Parameters
30B / 3B active
Context
1M
Attention
Mamba hybridinterleaved Mamba-2 and MoE layers with select GQA attention layers
Experts
128 routed + 1 shared, top-6
Input
text
Licence
OpenMDW-1.1
Released
2026-08-11

Against the consensus recipe

Its documents state 10 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 6 of the 19 architecture features checked — 4 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_nemotron_h.py, configuration_nemotron_h.py) and config.json at revision a9904d24bc, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention code only in its codenot stated

Full-attention layers project 32 query heads but only 2 key/value heads, which are repeated via repeat_kv for the group-wise attention…

show evidence

Full-attention layers project 32 query heads but only 2 key/value heads, which are repeated via repeat_kv for the group-wise attention computation.

num_attention_heads = 32num_key_value_heads = 2
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
        self.scaling = self.head_dim**-0.5
        self.attention_dropout = config.attention_dropout
        self.is_causal = True
        self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
        self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
modeling_nemotron_h.py · L877–882
Multi-head latent attention not in its codenot stated

There is no KV down-projection to a shared latent with per-head up-projection anywhere in the attention code; NemotronHAttention uses…

show evidence

There is no KV down-projection to a shared latent with per-head up-projection anywhere in the attention code; NemotronHAttention uses standard separate q/k/v projections.

self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
        self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
        self.v_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
modeling_nemotron_h.py · L881–883
Sliding-window attention not in its codenot stated

sliding_window is null in this config, so no attention layer restricts tokens to a local window.

show evidencesliding_window = null
Interleaved sliding-window and global attention not in its codenot stated

layers_block_type only distinguishes mamba/moe/attention (full_attention), with no sliding-window layer type or pattern present.

show evidencesliding_window = null
Indexer-selected sparse attention (DeepSeek Sparse Attention) not in its codenot stated

No indexer-scoring mechanism or top-k token selection code exists in the given files, and no index_topk/index_n_heads config keys are…

show the whole verdict

No indexer-scoring mechanism or top-k token selection code exists in the given files, and no index_topk/index_n_heads config keys are present.

Linear-attention or state-space layers alongside full attention code only in its codenot stated

layers_block_type interleaves mamba (linear_attention) layers with full_attention and moe layers across the 52-layer stack.

show evidencelayers_block_type = ["mamba", "moe", "mamba", "moe", "mamba", "attention", "moe", "mamba", "moe",…
MIXER_TYPES = {
    "linear_attention": NemotronHMamba2Mixer,
    "full_attention": NemotronHAttention,
    "moe": NemotronHMoE,
    "mlp": NemotronHMLP,
}
modeling_nemotron_h.py · L923–928
Gated DeltaNet layers not in its codenot stated

The recurrent layers implement the Mamba-2 selective SSM scan, not a gated delta rule; no delta-rule write or decay gate of that form…

show the whole verdict

The recurrent layers implement the Mamba-2 selective SSM scan, not a gated delta rule; no delta-rule write or decay gate of that form exists.

Mamba-2 layers in its codeadopted

The 'mamba' layers use NemotronHMamba2Mixer implementing the Mamba-2 SSD chunked scan with per-head scalar decay (A_log, D, dt_bias) and…

show evidence

The 'mamba' layers use NemotronHMamba2Mixer implementing the Mamba-2 SSD chunked scan with per-head scalar decay (A_log, D, dt_bias) and ssm_state_size.

mamba_num_heads = 64ssm_state_size = 128
self.A_log = nn.Parameter(torch.empty(self.num_heads))
        self.norm = NemotronHRMSNormGated(
            self.intermediate_size, group_size=self.intermediate_size // self.n_groups, eps=config.layer_norm_epsilon
        )
        self.D = nn.Parameter(torch.empty(self.num_heads))
modeling_nemotron_h.py · L440–444
Learnable attention sink not in its codenot stated

The eager/sdpa attention implementation has no extra learned sink logit column added to the softmax denominator.

show evidence
attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
    if attention_mask is not None:
        attn_weights = attn_weights + attention_mask

    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
modeling_nemotron_h.py · L856–860
Gated attention output not in its codenot stated

The attention output (NemotronHAttention) is passed directly to o_proj with no sigmoid gate multiplication; gating only appears in the…

show evidence

The attention output (NemotronHAttention) is passed directly to o_proj with no sigmoid gate multiplication; gating only appears in the Mamba mixer's silu gate, not in softmax attention.

attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights
modeling_nemotron_h.py · L918–920
QK normalization not in its codenot stated

NemotronHAttention projects q/k/v and applies no RMSNorm/LayerNorm to queries or keys before the dot product.

show evidence
query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
modeling_nemotron_h.py · L896–898
Partial RoPE code unclearnot stated

partial_rotary_factor is 1.0 (full rotary) but the attention forward code does not actually call apply_rotary_pos_emb or build rotary…

show evidence

partial_rotary_factor is 1.0 (full rotary) but the attention forward code does not actually call apply_rotary_pos_emb or build rotary embeddings in the given excerpt, so it's unclear whether/how rotary is applied at all for this model's attention layers.

partial_rotary_factor = 1.0
YaRN RoPE scaling not in its codenot stated

No rope_scaling key is present in this config, so no YaRN scaling is configured.

Layers without positional encoding (NoPE) code unclearadopted

The attention module's forward does not show rotary embedding application at all (no cos/sin passed into NemotronHAttention.forward), so…

show evidence

The attention module's forward does not show rotary embedding application at all (no cos/sin passed into NemotronHAttention.forward), so it cannot be determined whether some full-attention layers skip positional encoding while others apply it.

def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: torch.Tensor | None = None,
        past_key_values: Cache | None = None,
        **kwargs: Unpack[TransformersKwargs],
    ) -> tuple[torch.Tensor, torch.Tensor | None]:
modeling_nemotron_h.py · L886–892
Mixture of experts in its codeadopted

NemotronHMoE routes tokens via NemotronHTopkRouter to a subset (top-6 of 128) of NemotronHExperts.

show evidencen_routed_experts = 128num_experts_per_tok = 6
self.top_k = config.num_experts_per_tok
        self.num_experts = config.num_local_experts
        self.hidden_dim = config.hidden_size
        self.weight = nn.Parameter(torch.zeros(self.num_experts, self.hidden_dim))
modeling_nemotron_h.py · L760–763
Shared expert code only in its codenot stated

Each MoE layer includes a NemotronHMLP shared expert of intermediate size 3712, applied to every token in addition to routed experts, with…

show evidence

Each MoE layer includes a NemotronHMLP shared expert of intermediate size 3712, applied to every token in addition to routed experts, with n_shared_experts=1.

n_shared_experts = 1moe_shared_expert_intermediate_size = 3712
self.shared_experts = NemotronHMLP(config=config, intermediate_size=config.moe_shared_expert_intermediate_size)
modeling_nemotron_h.py · L731
hidden_states = hidden_states + self.shared_experts(residuals)
        return hidden_states
modeling_nemotron_h.py · L753–754
Auxiliary-loss-free load balancing (selection bias) code only in its codenot stated

The router adds e_score_correction_bias to the sigmoid scores used only for top-k expert selection, while the actual weights gathered for…

show evidence

The router adds e_score_correction_bias to the sigmoid scores used only for top-k expert selection, while the actual weights gathered for output weighting come from the unbiased scores.

scores = router_logits.sigmoid()
        scores_for_choice = scores + self.e_score_correction_bias
        group_scores = (
            scores_for_choice.view(-1, self.num_group, self.num_experts // self.num_group)
            .topk(2, dim=-1)[0]
            .sum(dim=-1)
        )
modeling_nemotron_h.py · L773–779
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_nemotron_h.py · L789–790
Multi-token prediction layers documents disagree not in its codeadopted

num_nextn_predict_layers is 1 in config but is not actually wired to build MTP modules in the given CausalLM/Model code (no mtp module…

show evidence

num_nextn_predict_layers is 1 in config but is not actually wired to build MTP modules in the given CausalLM/Model code (no mtp module construction appears, and loading explicitly ignores mtp.* keys), making the modules effectively absent from this build.

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"mtp.*"]
modeling_nemotron_h.py · L1012
SwiGLU feed-forward not in its codenot stated

Both the dense NemotronHMLP and the MoE expert MLP use a single activated up_proj followed by down_proj with relu2 activation (squared…

show evidence

Both the dense NemotronHMLP and the MoE expert MLP use a single activated up_proj followed by down_proj with relu2 activation (squared ReLU), not a SiLU-gated two-branch SwiGLU.

mlp_hidden_act = "relu2"
self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=config.mlp_bias)
        self.act_fn = ACT2FN[config.mlp_hidden_act]

    def forward(self, x):
        return self.down_proj(self.act_fn(self.up_proj(x)))
modeling_nemotron_h.py · L643–648

token mixer 5

Grouped dynamic convolution core

“each of the five layers of the drafter body wraps the attention and MLP sublayers in a grouped dynamic convolution”

softmax attention 1

sliding window attention 1

Sliding Window Attention with an 8,192-Position Window used

“We therefore serve it with sliding-window attention over the most recent 8,192 positions in every drafter layer”

linear attention & state space 1

Mamba 1

Mamba-2 core

“NVIDIA uses Mamba-2 for much of the sequence processing”

hybrid layer stacking 2

Hybrid Mamba-Transformer core

“both models combine large-scale distillation, long-context training, and native speculative decoding”

Hybrid Mamba-2 and Mixture-of-Experts Architecture core

“The model employs a hybrid Mixture-of-Experts architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention…”

channel mixer 5

mixture of experts 5

Mixture of experts core

“NVIDIA Nemotron 3.5 Lightning is an open 30B mixture-of-experts (MoE) model with 3B active parameters built for that execution layer of…”

Sparse expert activation core

“MoEs are fast and efficient because a router sends each token to just a few of its many experts, so only a fraction of the model’s…”

expert load balancing 1

Disabled router-bias updates used

“Router-bias updates are explicitly disabled.”

latent mixture of experts 2

LatentMoE core

“routed to an expanded set of experts that operate entirely in this latent space, and then projected back”

Latent-dimension reduction with increased expert count and active experts used

“we shrink the routed expert input dimension to reduce communication and memory costs, and reinvest the saved capacity into increasing the…”

positional encoding 1

No Position Encoding core

“Nemotron 3 models do not use RoPE in attention layers and therefore do not suffer from out-of-distribution RoPE issues during context…”

prediction head 7

Multi-Token Prediction core

“The model includes Multi-Token Prediction (MTP) layers, which predict multiple future tokens to provide richer training signals.”

Frozen-backbone MTP-head-only fine-tuning core

“the backbone remains frozen and only the MTP head parameters are updated.”

Recursively shared Multi-Token Prediction head core

“adopting the recursively shared MTP formulation”

Hidden-state mixing during recursive MTP training used

“input hidden states are sampled from the sequence of states produced by earlier MTP steps 1,...,k − 1, rather than relying exclusively on…”

Architecture-aware MTP-head initialization by projection fusion and pruning used

“We first fuse the latent MoE projections into the experts’ up- and down-projection matrices, then prune the hidden size from 4096 to 2688”

MTP-head warm-up with a frozen backbone used

“conduct a dedicated MTP warmup phase for 100 B tokens, freezing the backbone parameters and optimizing only the MTP head”

MTP-only training used

“We apply a dedicated MTP-only training recipe at two points”

filed at the root 1

Target-distribution calibration loss used

“We replace it with a calibration term.”

multi-token prediction objective 4

MTP Boosting used

“Post-training MTP boosting improves average speculative-decoding acceptance length”

Multi-Token Prediction used

“predicting multiple future tokens provides richer training signals and encourages models to plan several steps ahead”

Continued pretraining for MTP layers used

“NVIDIA added a dedicated continued-pretraining stage for these MTP layers.”

LK loss used

“Whereas Nemotron 3 Ultra uses a temperature-scaled forward-KL objective, we now use the LK loss”

distillation objective 8

Reverse KL divergence core

“The student minimizes the reverse KL𝐷KL(𝜋𝑆(·|𝑐𝑡)‖𝜋𝑇 (·|𝑐𝑡))at every position of its own responses”

Quantization-aware distillation used

“The released checkpoint uses the predefined aggressive PTQ recipe followed by SFT-based QAD at 200 iterations.”

Skew reverse-KL distillation used

“with Ultra 3 Base as the teacher and Skew Reverse KL divergence as the distillation loss function”

Backbone-to-MTP-head knowledge distillation used

“these runs use a knowledge-distillation loss with the frozen backbone as the teacher.”

Combined KL-divergence and cross-entropy distillation loss used

“we combined the KL loss with cross-entropy loss, assigning them weights of 0.9 and 0.1, respectively”

Forward-KL distillation with a sink token used

“We used standard forward KL divergence loss along with an extra sink token to capture the residual probability mass outside top-128.”

Full next-token distribution distillation used

“We depart from this recipe and, as in MTP boosting (Section 3.7), instead distill the full next-token distribution pM of the target model M”

KL-divergence distillation objective used

“We use KL divergence as our objective.”

auxiliary loss 2

Disable auxiliary and hard-target losses during QAD used

“We disable the main hard-target cross-entropy loss, MTP cross-entropy loss, and MoE load-balancing auxiliary loss and don’t use…”

Mask non-target tokens in rollout training data used

“We mask out prompt, conversation-history, tool-output, and prefilled response-prefix tokens.”

optimizer 1

Distributed AdamW used

“Optimization uses distributed AdamW with β1 = 0.9, β2 = 0.95, ϵ = 10−8, weight decay 0.1, and gradient clipping at a global norm of 1.0.”

learning-rate schedule 1

Linear learning-rate warmup followed by a constant rate used

“We reset the learning rate, warmed it up linearly to a maximum of1× 10−5, about twice the final pretraining learning rate, and then held…”

training precision 7

NVFP4 core

“The pre-training phase used an NVFP4 recipe.”

NVFP4 pretraining used

“The pre-training phase used an NVFP4 recipe.”

BF16 attention projections used

“we kept the QKV and attention projections in BF16”

FP32 language-model head used

“two optimizations designed to mitigate train–inference mismatch: router replay (Ma et al., 2025) and using FP32 for LM head (Chen et al.…”

High-precision final network layers used

“We kept the last 15% of the network in high precision to maintain stability.”

MXFP8 Mamba output projections used

“To prevent loss of information, we keep these layers in MXFP8.”

NVFP4 fine-grained micro-block scaling used

“The NVFP4 format features: fine-grained (16 element) micro-block scaling, block scaling factors with E4M3 format, a second-level FP32…”

quantization-aware training 6

Stochastic Rounding used

“such as randomness generation, stochastic rounding, or scaling calculations.”

Calibration Data used

“Activation scales are learned from a calibration set of image–text pairs, rather than text alone.”

Freezing Non-Target Parameters used

“All other parameters remain frozen, including token embeddings, unquantized language-model parameters, normalization parameters, the…”

NVFP4 Training used

“We demonstrate stable and accurate pretraining on a hybrid Mamba-MoE architecture for up to 25T tokens using the NVFP4 number format.”

Random Hadamard Transform used

“Random Hadamard Transforms (RHTs) on inputs to wgrad”

SFT-Based Quantization-Aware Distillation used

“The released checkpoint uses the predefined aggressive PTQ recipe followed by SFT-based QAD at 200 iterations.”

training stability 2

Global-norm gradient clipping used

“gradient clipping at a global norm of 1.0.”

Router replay used

“augmented with two optimizations designed to mitigate train–inference mismatch: router replay (Ma et al., 2025) and using FP32 for LM head”

training parallelism 3

Combined context, tensor, and expert parallel training used

“We used 16-way context parallelism, 8-way tensor parallelism, and 64-way expert parallelism to train on GB300 GPUs.”

Combined context, tensor, expert, and pipeline parallel training used

“We used 8-way context parallelism, 8-way tensor parallelism, 8-way expert parallelism, and 2-way pipeline parallelism to train on GB200…”

Data and expert parallelism for calibration used

“Pipeline parallelism left only one GPU active at a time, so we chose DP× EP.”

2 more mentioned or evaluated
BF16 expert activations evaluated

training precision “Runaway moves by ≈ 0; output verbosity falls on 5 of 5 focused benchmarks”

BF16 language-model head evaluated

training precision “No recovery of termination is captured.”

data sourcing 1

GitHub Crawl used

“The GitHub Crawl was collected using the GitHub REST API and the Amazon S3 API.”

data filtering 18

Benchmark decontamination used

“We therefore performed checks of post-training data against 30 benchmarks (see the complete list in Appendix C.2) and removed samples…”

Heuristic filtering used

“we applied just heuristic filtering instead—similar to what we did for lower quality English data in the Nemotron-CC pipeline”

Keyword- and regex-based content filtering used

“we apply targeted keyword- and regex-based filters and remove all trajectories matching such behavior.”

N-gram repetition filtering used

“We then aggressively filter reasoning traces exhibiting pathological repetition, such as repeated n-grams within a sliding window or…”

Permissive-license filtering used

“We collect raw source code and subsequently remove any having a license which does not exist in our permissive-license set.”

Unified data filtering pipeline used

“we apply a unified data filtering pipeline to ensure that only high-quality, license-compliant, and verifiable samples are used for…”

Automated quality filtering of synthetic vision data used

“we apply automated quality checks to filter ambiguous or visually unsupported questions, incomplete reasoning traces, hallucinated labels…”

Benchmark decontamination and secret removal from training sources used

“Every source is decontaminated against SWE-bench and Terminal-Bench and has secrets removed.”

Difficulty filtering for tool-dependent training examples used

“retaining samples that are nearly impossible to solve correctly without using the available tools.”

Minimum-length filtering for reasoning SFT data used

“we removed reasoning responses shorter than 5,000 tokens to focus the training mixture on examples containing sufficiently developed…”

N-gram verbatim-overlap detection used

“Verbatim overlap measures how many n-grams of the question’s text appear in the sample.”

Normalize training samples to their problem statements used

“The sample is reduced to its problem statement, the part into which a benchmark question could leak. Text templates that recur across its…”

Outcome- and execution-based filtering of agentic trajectories used

“We retain normally completed trajectories whose answers match the references and whose required tools execute successfully.”

Per-question retrieval of contamination candidates used

“per question: top 20 by combined score+top 2 per signal”

Reference-answer compatibility auditing used

“We audit reference-answer formats and compatibility with the reward functions, and remove exact question–answer duplicates from the…”

Severity-thresholded benchmark contamination removal used

“A sample is removed if the judge finds it severely contaminated with respect to any benchmark question.”

Structural checks for malformed examples used

“We first discard malformed examples using structural checks (e.g., missing tool definitions when tool calls are present).”

Translation quality-score threshold filtering used

“We keep translations scoring at least 75.”

deduplication 2

Corpus deduplication used

“Deduplication was done in the same way as for Nemotron-CC.”

Exact question–answer deduplication used

“remove exact question–answer duplicates from the second-stage mixture.”

synthetic data 18

Knowledge distillation for synthetic data generation used

“we generate synthetic data by distilling trajectories, solutions, and translations from strong teacher models and agent systems, often…”

Agentic visual tool use for image operations used

“We use Kimi K2.6 (Kimi Team, 2025) and K3 (Kimi Team, 2026) to generate agentic datasets with a focus on image tool operations such as…”

Bodhan-MT translation with Gemma-4 quality assessment and language-specific thresholds used

“For the ten additional lower-resource languages, we use Bodhan-MT for translation and Gemma-4-31B-it for quality assessment, with…”

Chunked synthetic long-context translation data used

“we synthetically generate long-context translation data from both the PG19 (Rae et al., 2019) and FinePDFs-Edu (Kydlíček et al., 2025)…”

Concurrent scheduling of translation and quality checks used

“we improved scheduling so translation and quality checks can run concurrently.”

Coverage-audit-driven agentic question generation used

“The agentic track targets gaps found in a coverage audit against the Vals.ai category taxonomy by making the relationship among source…”

Document-grounded synthetic question generation with answerability verification used

“The document-grounded track samples windowed contexts from SEC 10-K, 10-Q, and 8-K filings of S&P 500 companies, generates analyst-style…”

Four-stage Indic synthetic-data generation pipeline used

“We implement the four-stage pipeline for the twelve priority languages”

Large-scale synthetic data generation, filtering, and curation used

“Built with large-scale synthetic data generation, filtering, and curation”

Overlapping windows for longer Santali documents used

“We use overlapping windows only for longer Santali documents”

Rewrite low-scoring translations using source text and identified errors, then recheck used

“Lower-scoring translations are rewritten using the source text and identified errors, then checked again and kept only if they pass.”

Score translations against their source for quality used

“The model then checks each translation against its source and assigns a quality score from 0 to 100.”

Step-structured, visually grounded reasoning traces used

“organizing the reasoning traces into an explicit sequence of concise steps, with each step grounded in relevant visual evidence, helps…”

Synthetic graph-structured long-context reasoning data used

“we synthetically generated graph-structured data to teach the model to trace nodes and edges through reasoning, rather than relying solely…”

Synthetic Indic training-data generation by translating curated English documents used

“We address these gaps by translating curated English documents into Indic languages.”

Synthetic privacy-preserving personas used

“Fully synthetic, privacy-preserving personas grounded in real-world demographic, geographic, and cultural distributions”

Synthetic transduction-task training for long translation used

“long-translation performance was improved by also including synthetic transduction tasks (Willette et al., 2026) that reinforce position…”

Translate each selected source document in full without splitting used

“translate each document in full, without splitting”

data mixture & curriculum 6

Curated internal-data mixture for vision tool-use training used

“we train a vision tool-calling teacher on a curated mixture of internal data.”

Difficulty-based routing of validated questions to SFT or RL used

“are routed to the SFT or RL pool according to their measured difficulty.”

Indic continued-pretraining data allocation used

“10% (10B tokens) allocated to selected synthetic text from our corpus and natural Indic text.”

Preserve natural training-data format used

“Most of it is natural data kept in its original format, without chat templates.”

Reasoning-focused RL curriculum used

“We replace the broad RLVR mixture with a reasoning-focused curriculum of 25,600 examples, with each 256-prompt batch containing 90%…”

Stage-specific domain weighting by distillation learning curves used

“stage 1 is weighted toward the domains that improve quickly and tend to regress later, and stage 2 toward the domains that improve more…”

3 more mentioned or evaluated
Fixed-budget Indic mixture-ratio ablations evaluated

data mixture & curriculum “we ran separate Lightning 3.5 research experiments with 10%, 20%, and 30% Indic data under a fixed 200B-token training budget.”

Mix supervised fine-tuning data with teacher-generated reinforcement-learning rollouts evaluated

data mixture & curriculum “a mixture of 75% SFT data with 25% teacher-generated RL rollouts”

Counterfactual data augmentation mentioned

synthetic data “mitigation strategies like counterfactual data augmentation to align with the desired model behavior”

filed at the root 1

Specialized teacher training used

“We further train specialized teachers for video reasoning and tracking, vision tool use, and alignment”

supervised fine-tuning 9

Supervised fine-tuning used

“The model was further fine-tuned on synthetic code, math, science, tool calling, instruction following, structured outputs, and general…”

Low-Rank Adaptation used

“Establish a baseline, train and deploy a LoRA adapter, and evaluate task-specific improvements in minutes”

Two-stage supervised fine-tuning used

“we use a two-stage SFT process because learning vision and text capabilities simultaneously is challenging.”

Capability-targeted second-stage supervised fine-tuning used

“we do a small second stage of SFT, where we change the blend composition to boost some capabilities the first stage did not learn…”

Mask tool responses from the SFT loss used

“The agent traces are the only SFT-style data, so we keep them to a small share of the blend and mask tool responses from the loss…”

Random-budget truncation of reasoning traces used

“the reasoning traces of a small fraction of SFT data are truncated to random reasoning budgets while the responses remain the same and…”

Single-stage supervised fine-tuning used

“Nemotron 3.5 Lightning is trained with single SFT stage with the text only SFT data.”

Supervised fine-tuning at 256k sequence length used

“supervised fine-tuning (SFT) was performed at a 256k sequence length”

Use 100% SFT data for final QAD used

“we reduced checkpointing interval to 100 iterations, used the 100% SFT blend only, and evaluated QAD checkpoints at 200, 300, and 400…”

reinforcement learning algorithm 8

Group Relative Policy Optimization used

“The model underwent multi-environment reinforcement learning using GRPO (Group Relative Policy Optimization) across math, code, science…”

Reinforcement learning from verifiable rewards used

“We post-train Nemotron 3.5 Super through a comprehensive pipeline spanning Supervised Fine Tuning (SFT), Reinforcement Learning from…”

Asynchronous Group Relative Policy Optimization used

“We initialize from an existing multimodal RLVR checkpoint and train using asynchronous GRPO”

Abstention training used

“We mix this RLHF training with the abstention training strategy from Nemotron 3 Ultra”

Clipped-range importance weighting with stale-token masking used

“tokens are then importance-weighted by¯𝜋𝑆/¯𝜋𝑆−1 and those with weights outside[0.2, 5]are masked”

End-to-end GRPO on agentic tasks used

“continue training with full end to end GRPO on agentic tasks.”

GRPO with masked importance sampling used

“we use GRPO (Shao et al., 2024) with masked importance sampling to account for discrepancies between the training and rollout policies”

Length-based RL reward adjustment used

“length-based adjustments are applied on the RL rewards for them.”

reward modelling 8

Reinforcement Learning with Verifiable Rewards used

“Use reinforcement learning with verifiable rewards to adapt NVIDIA Nemotron for specific domains and workflows”

Reasoning-length penalty used

“First, we train a new variant of the Nemotron 3 Ultra GenRM that is trained with a reasoning length penalty to be more token efficient.”

Auxiliary tool-use reward used

“as well as an auxiliary tool-use reward to penalize duplicate and malformed tool calls.”

Binary answer-correctness reward used

“Video-QA uses a binary reward based on multiple-choice answer correctness.”

Continuous tracking-quality reward used

“the verifier assigns a continuous reward based on localization quality, object-visibility accuracy, and the validity and completeness of…”

Correct-response-only soft length preference used

“Second, we apply a soft length preference only to correct responses.”

Deterministic task-verifier rewards used

“Both sources use deterministic, verifiable task rewards rather than an LLM judge.”

Matched think-block format validation used

“a response without exactly one properly matched <think>...</think> block is considered malformed: its task reward is set to zero and its…”

preference optimization 2

Reinforcement Learning from Human Feedback (RLHF) core

“We train this teacher primarily using RLHF.”

Fixed-reference response comparison used

“we instead compare all policy responses to a fixed response from a strong open-source model, Gemma 4 31B-it.”

policy distillation 6

Token-Local Distillation Update core

“using the token-local update”

Multi-Teacher On-Policy Distillation used

“After MOPD, truncation decreases to 0.3% with 13,669 output tokens per rollout, where only one runaway loop failure is captured.”

Distillation from a Larger Nemotron 3 Model used

“then distilled each model for 1 trillion tokens”

Frozen Student Self-Anchor used

“environments no specialist owns are routed to a self-anchor, a frozen copy of the student’s starting checkpoint”

Proximal-Teacher Trust-Region Policy Distillation used

“the distillation target is smoothed with the proximal-teacher construction of trust-region policy distillation”

Two-Stage Multi-Teacher On-Policy Distillation used

“MOPD for Nemotron 3.5 Super is run in two stages on 128 GB200 nodes.”

rollout & RL infrastructure 5

Asynchronous reinforcement learning used

“It utilized an asynchronous RL architecture that decouples training from inference and leverages MTP to accelerate rollout generation.”

On-policy rollout generation used

“We generate on-policy rollouts using the corresponding backbone checkpoint.”

Independent parallel agent rollouts used

“For each task, N independent instances of the same post-trained agent attempt the task, producing a pool of candidate trajectories. These…”

One-step-stale asynchronous response generation used

“responses may come from the previous snapshot¯𝜋𝑆−1, i.e. one step stale”

Teacher-generated rollout trajectories used

“For the RL rollouts we sample 16 trajectories per source record with vLLM.”

agentic post-training 4

Multi-environment reinforcement learning core

“we train the Nemotron 3 models on all of these tasks simultaneously”

Finance-agent environment with consistent tools across training stages used

“The environment provides the same tools—SEC filing search, document parsing, evidence retrieval, and final source-supported…”

Harness-optimized training used

“Harness-optimized training: The model is trained for popular agent harnesses, enabling agents to make more accurate calls while reducing…”

Training agentic teachers on sandboxed executable tasks used

“opting to train the end to end agentic teacher on sandbox, fully executable agentic Terminal, SWE, and Search tasks.”

mid-training & continual pretraining 5

Long-context extension via continued pretraining core

“In the LC-Phase, we performed continued pretraining (CPT) at 1M-token context length to equip the base model with long-context ability.”

Mid-training core

“Mid-training sits between the two. It trains the base model at long context on a diverse mixture of natural software-engineering data with…”

Indic continued pretraining used

“combines synthetic text generated from English documents with natural Indic corpora to strengthen Indian-language capabilities”

Long-context recovery training used

“To recover long-context capabilities affected by 256K mid-training, we perform an additional 9 billion token recovery stage”

Software-engineering mid-training used

“The mid-training mixture combines whole repositories, issue–pull-request pairs, commit histories, build and CI logs, and agent trajectories.”

2 more mentioned or evaluated
Experimental agentic SFT blend evaluated

supervised fine-tuning “This SFT is an experimental agentic blend of about 33B tokens (2K steps), not the one used in Section 3”

Fine-tuning with demographically balanced datasets mentioned

supervised fine-tuning “fine-tuning with demographically balanced datasets”

filed at the root 1

Platform- and workload-specific quantization and speculative-decoding tuning used

“Rather than impose one serving configuration everywhere, we tune quantization and speculative decoding for each platform.”

decoding strategy 10

Speculative decoding core

“This is a process whereby the model, or draft model, will draft some number of tokens which are efficiently reviewed.”

Top-K candidate lattice core

“LiLiCorr retains the K most probable tokens of the drafter body’s marginal as candidates”

Multi-Token Prediction used

“the system can speculate about: token t+1 token t+2 token t+3 ... Those candidates can then be verified efficiently.”

Best-of-N selection used

“candidate solutions, best-of-n selection traces, and verified CUDA kernels”

Standalone parallel speculative drafters used

“parallel drafters propose all draft tokens in a single forward pass whose latency is nearly independent of the draft length.”

LiLiCorr single-pass correlation head used

“LiLiCorr instead uses a lightweight, single-pass correlation head to score compatibility across slots.”

Semi-autoregressive drafting with sequential Markov correction used

“It keeps the drafter body parallel and adds a lightweight sequential head that biases the logits at each slot using a low-rank transition…”

DSpark optional

“DSpark: A semi-autoregressive speculative-decoding drafter that proposes a whole block of candidate tokens in a single forward pass from a…”

DFlash optional

“DFlash: A speculative-decoding drafter that uses a lightweight block-diffusion model to generate an entire draft block in one forward pass.”

Probabilistic draft sampling optional

“Under probabilistic draft sampling, each slot is instead drawn from the head’s distribution over its candidates at the request’s…”

reasoning control 1

Reasoning token-budget control used

“the Nemotron 3 models are trained to work with inference-time budget control”

KV cache management 4

Hybrid-state caching used

“We additionally optimize hybrid-state caching and GPU kernels for efficient long-context and agentic serving”

Internal prefill checkpointing used

“The adapted path arranges for a logical chunk to end at the desired cache boundary, extracts the state and convolution window there, and…”

Checkpointing replay for Mamba recurrent-state cache used

“We have since implemented a "checkpointing replay" method.”

Mamba checkpointing for speculative-decoding replay optional

“We improved this path by optionally retaining an additional Mamba cache page near the speculative-decoding replay boundary.”

inference quantization 20

Backbone-coupled MTP KV-cache precision core

“MTP KV precision therefore follows the backbone.”

Shared FP8 lm_head for target verification and MTP drafting default

“thus a single FP8 lm_head serves both target verification and MTP drafting”

Post-training quantization used

“We apply post-training quantization (PTQ) and quantization-aware distillation (QAD) with ModelOpt on Lightning 3.5 and Super 3.5 VL”

MSE-based weight-scale selection used

“Hence we use the MSE calibration for the PTQ and AutoQuant recipes.”

Aggressive mixed-precision quantization recipe used

“The released checkpoint uses the aggressive recipe, with NVFP4 MoE and shared-expert GEMMs and FP8 attention projections and Mamba linear…”

BF16 MTP attention weights with FP8 K/V BMM quantizers used

“The MTP attention weights remain BF16, with only the K/V BMM quantizers at FP8.”

FP8-or-higher lm_head precision used

“To maintain token generation verbosity, the lm_head should stay at FP8 weights or above”

Gradient-based per-layer mixed-precision quantization search used

“the mixed-precision checkpoints of §4.3, produced by a gradient-based per-layer search”

NVFP4 post-training quantization used

“We develop NVFP4 post-training quantization and mixed-precision recipes across weights, caches, and the MTP drafter”

NVFP4 W4A4 quantization of MTP experts used

“We therefore quantize the MTP experts using NVFP4 W4A4.”

NVFP4 weight quantization with BF16 activations used

“quantizing LiLiCorr’s linear layers to NVFP4 while keeping the activations at BF16, resulted in a 11% throughput improvement on DGX Spark”

Parameter alignment for quantization used

“To address this issue, we preferred parameter alignment as described in §4.4.”

Per-block FP8 quantization for attention Q/K/V projections used

“This study recommends FP8 precision on 𝑞/𝑘/𝑣 projection at per-block scale granularity.”

Post-training calibration sample count used

“We use the SFT data employed during post-training as the calibration corpus and calibrate the model using 1,024 samples.”

Quantization-aware distillation used

“complemented by quantization-aware distillation, achieving 99.62% median accuracy recovery relative to BF16”

Separate MTP quantization and backbone splicing used

“The released checkpoints were produced by quantizing the boosted MTP separately and splicing it into the selected quantized backbone”

Task-loss-gradient-weighted quantization sensitivity estimation used

“AutoQuant estimates a sensitivity Sᵢ(q) from the quantization-induced output error weighted by the squared task-loss gradient.”

Weight-only post-training quantization for parallel drafters used

“we applied weight-only PTQ to improve the drafting efficiency of a DSpark module for Nemotron Lightning 3.5 and LiLiCorr module for…”

NVFP4 quantization optional

“Nemotron 3.5 Lightning ships with an NVFP4 checkpoint alongside BF16, using the same specialized NVFP4 kernels that power Nemotron 3 Ultra…”

Separate target and MTP lm_head precision optional

“This allows the target lm_head to remain in BF16 while using a lower-precision lm_head for drafting.”

inference scheduling 2

KV-cache-aware routing used

“A KV-cache-aware router sends each request to an instance.”

Session affinity routing used

“Session affinity sends all turns of one session to the same worker, so later turns find their prefix in its cache.”

inference kernel 3

GPU-kernel optimization used

“optimize hybrid-state caching and GPU kernels for efficient long-context and agentic serving”

Multi-wave KV-cache splitting optional

“The multi-wave KV split then splits the KV cache further and runs the work in several waves.”

Per-quantization-scheme GEMM backend selection optional

“The global linear backend remains the default, while a kernel configuration can override it for a specific quantization scheme”

agentic scaffolding 3

Model routing by planning and execution role core

“Plans route up to the frontier, execution routes down to Lightning, ensuring that your tokens are spent efficiently and effectively.”

Parallel agent execution with self-verification used

“We further improve the final model’s agentic performance through parallel agents followed by self-verification.”

Bash computer-use agent used

“translates natural language requests into safe, human-approved terminal commands using tool calling, execution feedback, and optional…”

20 more mentioned or evaluated
Disaggregated serving evaluated

serving parallelism “The disaggregated deployment uses 7 GPUs for prefill only and 4 GPUs for decode only.”

Parallel test-time compute scaling evaluated

reasoning control “we explore test-time scaling through parallel agent execution and self-verification, improving Terminal-Bench 2.1 from 62.7% to 71.9%”

Per-tensor FP8 quantization evaluated

inference quantization “aggregation benchmarks can identify important candidates in FP8, where even per-tensor FP8 precision can be acceptable over 100 tasks at…”

BF16 expert activations evaluated

inference quantization “Expert activations → BF16. Runaway moves by ≈ 0”

Block-scaled NVFP4 KV-cache quantization evaluated

inference quantization “NVFP4 stores 4-bit E2M1 values with one FP8 E4M3 scale per 16-element block”

Concurrency-aware draft-length tuning evaluated

decoding strategy “MTP is best suited for medium to high concurrency, with the optimal draft length decreasing as concurrency increases.”

FP4 quantization of Mamba output-projection weights evaluated

inference quantization “quantizing its weights to FP4 inflates generation length.”

FP8 or NVFP4 KV-cache quantization evaluated

inference quantization “in each case both the key and value caches across attention layers are quantized in FP8 or NVFP4, respectively.”

Increase MoE expert precision and fine-tune on RL rollouts evaluated

inference quantization “Empirically, promoting MoE expert precision and fine-tuning on RL rollouts can mitigate this effect”

Linear E2M1 residual correction evaluated

inference quantization “with a linear E2M1 residual the KL falls to 0.00117 against per-tensor FP8’s 0.00134”

MXFP8 block-scaled attention projection quantization evaluated

inference quantization “Substituting MXFP8 for per-tensor FP8 on 𝑞/𝑘/𝑣 changes only the scale granularity with one E8M0 exponent per 32 elements instead of one…”

Promote expert weights from NVFP4 to FP8 evaluated

inference quantization “Routed/shared experts NVFP4→ FP8. In tau3-banking, runaway dropped from 28.2% → 17.9%”

Quantization of the parallel-drafter language-model head evaluated

inference quantization “Further quantizing the LM head improved throughput by another 5%.”

Restricted-search MSE weight-scale selection evaluated

inference quantization “MSE 4-over-6: MSE-based scale selection restricted to a narrower search space.”

Rotation or higher precision for Mamba input activations evaluated

inference quantization “Rotation or higher precision can mitigate this effect in activation”

Sensitivity-driven mixed-precision quantization evaluated

inference quantization “AutoQuant was explored as an alternative sensitivity-driven mixed-precision approach.”

Two-pass FP4 weight quantization with residual correction evaluated

inference quantization “we emulated two-pass FP4 weights, where a second channel corrects the residual left below the amax/24 flush line.”

W4A16 quantization of Mamba output-projection weights evaluated

inference quantization “output verbosity increased by 18.5% at 2.4𝜎, while the runaway rate and knowledge abstention were both unaffected.”

Higher-precision activations for Mamba output projection mentioned

inference quantization “Applying rotation or higher activation precision can mitigate the flush.”

Rotation-based quantization mitigation mentioned

inference quantization “Applying rotation or higher activation precision can mitigate the flush.”

Not used 3

Confidence head not used

decoding strategy “The Lightning 3.5 release used the lightweight Markov correction but omitted this confidence head.”

Mixed-precision key/value cache quantization not used

inference quantization “Per-layer mixed FP8-key/NVFP4-value (or vice-versa) configurations are not evaluated in this comparison.”

W4A4 quantization not used

inference quantization “This tensor analysis stops us from taking the Mamba output projection to W4A4.”

filed at the root 2

Nemotron Parse used

“Understands document semantics and extract text and tables elements with spatial grounding”

Model routing and orchestration optional

“Model routing and orchestration help make this division of labor more accessible. They are now available through NVIDIA NeMo Switchyard.”

inference engine 1

TensorRT-LLM used

“TensorRT™-LLM is an open source library built to deliver high-performance, real-time inference optimization for large language models”

infrastructure service 2

NeMo Guardrails used

“NeMo Guardrails, a flexible, open library for defining and enforcing enterprise AI policies in real time”

NeMo Switchyard used

“It routes prompts to the most capable and efficient model for each step of an agent workflow automatically, based on configurable routing…”

1 more mentioned or evaluated
ModelOpt AutoQuantize evaluated

kernel & quantization library “We use ModelOpt AutoQuantize (AutoQuant) to replace hand-authored layer rules with a sensitivity-constrained mixed-precision search.”

filed at the root 11

Benchmark decontamination core

“The benchmark side (question store, n-gram index, and embedding index) is built once and shared by all runs.”

Chain-of-thought few-shot prompting used

“We evaluate MGSM (Shi et al., 2022) using 8-shot native chain-of-thought prompting in Spanish, German, French, Chinese, Japanese, and…”

Controlled evaluation with fixed settings used

“All models were evaluated under the same evaluation settings including agentic resources (CPU compute and timeouts, tools, search APIs)…”

Candidate-pair overlap gate used

“Judge gate semantic overlap≥0.75 or verbatim overlap≥0.5”

Capability-partition stratification used

“It is important to stratify test cases along with capability partition to assess quantization damage properly.”

Context-length and candidate-count stratification used

“Table 35 stratifies 1,470 MRCR-128k tasks at three repetitions to separate the retrieval from decoding noise.”

Hard same-domain negative examples used

“As negatives we add unmodified questions from a different benchmark in the same domain, chosen to be hard to tell apart from the target”

Overlapping byte-window chunking used

“Chunking 512-byte windows with 256-byte stride (sample and question)”

Rollout non-termination tracking used

“while tracking the failure of a model to terminate as it runs to the context wall”

Separate augmentation and judge models used

“The LLM augmentations are generated with a model different from the judge, so that the judge cannot recognize its own rewrites.”

Synthetic benchmark-question perturbation evaluation used

“transform each with 13 augmentations, at four strengths (0.25, 0.5, 0.75, 0.9)”

benchmark 5

EvalPlus-sanitized benchmark variants used

“using EvalPlus-sanitized variants (Liu et al., 2023).”

Five-shot chain-of-thought prompting for GPQA Diamond used

“GPQA Diamond is evaluated using 5-shot chain-of-thought prompting.”

Three-shot prompting for MBPP-Sanitized used

“MBPP-Sanitized uses 3-shot prompting”

Verified CUDA kernels for synthetic data used

“candidate solutions, best-of-n selection traces, and verified CUDA kernels”

Zero-shot prompting for HumanEval used

“while HumanEval uses zero-shot prompting.”

evaluation harness 3

NeMo Evaluator SDK used

“All evaluation results reported for Nemotron 3.5 Super 120B-A12B Base and Lightning 3.5 30B-A3B Base were collected via NeMo Evaluator SDK”

LM Evaluation Harness used

“were collected via NeMo Evaluator SDK and NVIDIA’s open-source container of LM Evaluation Harness”

NeMo Gym and NeMo Evaluator harness used

“NVIDIA says these evaluations were run through a consistent NeMo Gym and NeMo Evaluator-based harness”

judge 8

Claude Opus 4.6 judge used

“Judge model Claude Opus 4.6, temperature 0”

JSON-escaped input formatting for judge used

“Passing both texts as JSON-escaped strings keeps their boundaries unambiguous even when the sample itself contains code, markup, or…”

LLM-as-a-Verifier used

“The same checkpoint then serves as a verifier, selecting one candidate using LLM-as-a-Verifier (LAV).”

LLM-judge verification of contamination pairs used

“Each shortlisted pair of a sample and a benchmark question is verified by an LLM judge.”

Order-swapped verifier evaluation used

“For each compared pair and criterion, we evaluate both presentation orders”

Probabilistic Pivot Tournament used

“These pairwise evaluations are organized using LAV’s Probabilistic Pivot Tournament (PPT).”

Teacher-based text-only equivalence judging used

“other free-form answers require same-teacher, text-only equivalence judging with reported confidence≥ 0.90.”

Three-criterion trajectory verification used

“evaluates them against three criteria: specification adherence, output match, and error signal detection.”

averaging - mean@k; avg@k; pass@k; pass^k 3

4-shot sampled pass@1 used

“We additionally evaluate MATH-500 using 4-shot sampled generation with 32 generations per prompt and report pass@1.”

Mean of per-rubric scores used

“We also use the mean of per rubric scores to evaluate responses instead of an overall score.”

Sampled pass@1 used

“We report sampled pass@1 estimated from 32 generations per prompt.”

3 more mentioned or evaluated
Artificial Analysis Intelligence Index evaluated

benchmark “This index combines nine evaluations to measure model performance across agentic tasks, coding, scientific reasoning, and general…”

Matched supervised fine-tuning checkpoint comparison evaluated

“To measure the effect of mid-training, we ran identical SFT on the base checkpoint and on the mid-trained checkpoint and compared the two.”

V-model testing and validation mentioned

“Following the V-model methodology, iterative testing and validation at both unit and system levels are essential”

filed at the root 1

Postmortem reasoning review used

“We identify the runaway loop by reading reasoning passages in postmortem”

1 more mentioned or evaluated
Ablation of non-attention components evaluated

“We divide and conquer the degradation by ablating non-attention components in pre-MOPD models”

unfiled 23

Joint Transformer processing of the LiLiCorr candidate lattice core

“processes the resulting (B−1)×K candidate lattice jointly with a small Transformer”

Model specialization for agent execution core

“use the expensive model to think and the fast model to work”

Configurable routing profiles for model selection used

“using request signals and routing policies to send each agent task to the right model”

Convert Manipuri translations from Bengali script to Meitei Mayek used

“convert Manipuri translations from Bengali script to Meitei Mayek”

Dual-signal benchmark contamination candidate retrieval used

“The sample is compared with every benchmark question using two signals.”

Embedding-based semantic-overlap detection for benchmark contamination used

“Semantic overlap maximum cosine similarity over chunk pairs”

Embedding-similarity semantic contamination detection used

“Semantic overlap measures the embedding similarity between parts of the two texts.”

Executable visual-tool-use demonstrations for video understanding used

“generate executable SpatialClaw (Cho et al., 2026) demonstrations from existing video question–answer pairs, teaching the student to…”

frontier/execution model division of labor used

“Frontier reasoning models such as Nemotron 3 Ultra handle orchestration and complex planning while smaller, more efficient models handle…”

Left-to-right greedy candidate selection by adjacent-slot compatibility used

“the block is committed from left to right, each slot selecting the candidate most compatible with its committed predecessor”

Measure output length per rollout as a runaway-loop indicator used

“The combination of these two quantities is a useful early-warning indicator.”

Measure rollout truncation rate used

“truncation, the fraction of rollouts containing at least one model call hitting an output limit”

OpenAI-compatible API endpoint used

“NVIDIA exposes the model through an OpenAI-compatible endpoint.”

Quantizer-enabled language-model GEMM-only QAD updates used

“We optimize only the weights of language-model GEMMs with enabled quantizers. All other parameters remain frozen”

Reinforcement learning datasets with trajectories, tool calls, and preference signals used

“including multi-turn trajectories, tool calls, and preference signals across coding, math, reasoning, and agentic tasks”

Replay rollout-time MoE routing decisions during policy evaluation used

“We replay the MoE expert-routing decisions recorded during rollout generation when evaluating the policy, reducing train–generation…”

Reproducible evaluation recipes in NeMo Gym used

“the evaluation recipes, installation instructions, and commands for NVIDIA Nemotron 3.5 Lightning were collected and published in NeMo Gym”

Restore PTQ quantizer configuration and statistics for QAD used

“The quantization configuration and calibrated quantizer statistics are restored from the PTQ checkpoint rather than recalibrated during QAD.”

Restrict QAD optimization to quantized language-model GEMM weights used

“We optimize only the weights of language-model GEMMs with enabled quantizers.”

Select QAD training duration by evaluation rather than assuming more iterations help used

“Overall, our results favor shorter QAD training with 100% SFT data and show that optimal training duration depends on the selected data…”

Speculative decoding, FP8 weights, and FlashAttention for translation efficiency used

“Speculative decoding, FP8 weights, and FlashAttention reduced median translation cost”

task-to-best-model routing with NeMo Switchyard optional

“Switchyard can expose Nemotron 3.5 Lightning as a routing target alongside your open and closed models, so every request lands on the most…”

1 more mentioned or evaluated
Bias audits mentioned

“we recommend considering evaluation techniques such as bias audits”

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

↑