Open Models Almanac
ModelsMistral-Large-4

Mistral AI

Mistral-Large-4

What Mistral-Large-4’s own documents say it is built from — every method with the sentence that describes it.

Parameters
~1T / ~52B active
Context
1M
Attention
sparse (indexer)MLA converted to DeepSeek Sparse Attention (indexer top-k 2048), NoPE every 4th layer, YaRN
Experts
128 routed + 1 shared, top-4 (first layer dense)
Input
text, image
Licence
Mistral Ecosystem Open-Weight License (modified MIT)
Released
2026-10-29

Against the consensus recipe

Its documents state 8 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 8 of the 19 architecture features checked — 4 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_mistral_large4.py, configuration_mistral_large4.py) and config.json at revision af26e7869a, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention not in its codenot stated

num_attention_heads equals num_key_value_heads (64 both) and attention uses MLA-style latent KV rather than GQA with fewer KV heads.

show evidencetext_config.num_attention_heads = 64text_config.num_key_value_heads = 64
Multi-head latent attention in its codeadopted

Keys/values are compressed into a shared kv_lora_rank latent via kv_a_proj_with_mqa/kv_a_layernorm and up-projected per head by kv_b_proj…

show evidence

Keys/values are compressed into a shared kv_lora_rank latent via kv_a_proj_with_mqa/kv_a_layernorm and up-projected per head by kv_b_proj, with a decoupled RoPE key, DeepSeek-V2 style.

text_config.kv_lora_rank = 512text_config.qk_rope_head_dim = 64
self.kv_a_proj_with_mqa = (
            nn.Linear(self.hidden_size, config.kv_lora_rank + self.qk_rope_head_dim, bias=config.attention_bias)
            if not self.is_kv_shared_layer
            else None
        )
modeling_mistral_large4.py · L350–354
self.kv_b_proj = nn.Linear(
            config.kv_lora_rank,
            self.num_heads * (self.qk_nope_head_dim + self.v_head_dim),
            bias=False,
        )
modeling_mistral_large4.py · L358–362
Sliding-window attention not in its codenot used

No sliding_window key or windowed masking mechanism appears in the config or code; all layers are 'indexed_attention' with full causal…

show the whole verdict

No sliding_window key or windowed masking mechanism appears in the config or code; all layers are 'indexed_attention' with full causal masking.

Interleaved sliding-window and global attention not in its codenot stated

layer_types lists only 'indexed_attention' for every layer, with no sliding-window variant interleaved.

Indexer-selected sparse attention (DeepSeek Sparse Attention) in its codeadopted

A MistralLarge4TextIndexer scores past tokens and the attention forward selects the indexer's top-k (index_topk) token indices to restrict…

show evidence

A MistralLarge4TextIndexer scores past tokens and the attention forward selects the indexer's top-k (index_topk) token indices to restrict attention, DeepSeek Sparse Attention style.

text_config.index_topk = 2048text_config.index_n_heads = 32text_config.index_head_dim = 128
topk = min(self.index_topk, index_scores.shape[-1])
        return index_scores.topk(topk, dim=-1).indices.to(torch.int32)  # [B, S, topk]
modeling_mistral_large4.py · L273–274
if self.indexer is not None:
            topk_indices = self.indexer(
                hidden_states,
                position_embeddings,
                attention_mask[:, 0, :, :],
                position_ids,  # Kept for BC
                past_key_values=past_key_values,
            )  # [B, S, topk]
modeling_mistral_large4.py · L447–454
Linear-attention or state-space layers alongside full attention not in its codenot stated

There is no linear-attention or SSM mixer implementation; every layer is the same MLA/DSA attention module per layer_types.

Gated DeltaNet layers not in its codenot stated

No gated delta-rule recurrent mixer or related config keys (linear_num_key_heads, linear_conv_kernel_dim) exist in this model's code or…

show the whole verdict

No gated delta-rule recurrent mixer or related config keys (linear_num_key_heads, linear_conv_kernel_dim) exist in this model's code or config.

Mamba-2 layers not in its codenot stated

No Mamba-2/SSD state-space block implementation or related config keys (mamba_num_heads, ssm_state_size) are present.

Learnable attention sink not in its codenot stated

There is no learned per-head sink logit added to the attention softmax denominator anywhere in the attention or eager_attention_forward…

show the whole verdict

There is no learned per-head sink logit added to the attention softmax denominator anywhere in the attention or eager_attention_forward code.

Gated attention output not in its codenot stated

The attention output is reshaped and passed directly to o_proj with no sigmoid gating multiplication.

show evidence
attn_output = attn_output.reshape(batch_size, seq_length, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights, topk_indices
modeling_mistral_large4.py · L491–493
QK normalization not in its codenot stated

Queries and keys in the main attention module are not normalized before the dot product; only the indexer's keys use a LayerNorm (k_norm)…

show the whole verdict

Queries and keys in the main attention module are not normalized before the dot product; only the indexer's keys use a LayerNorm (k_norm), which is part of the separate sparse indexer, not the main QK path.

Partial RoPE code only in its codenot stated

RoPE is applied only to the qk_rope_head_dim slice (64) of each 128-dim qk head while qk_nope_head_dim (64) carries no rotary signal…

show evidence

RoPE is applied only to the qk_rope_head_dim slice (64) of each 128-dim qk head while qk_nope_head_dim (64) carries no rotary signal, DeepSeek-style decoupled RoPE.

text_config.qk_rope_head_dim = 64text_config.qk_nope_head_dim = 64
q_pass, q_rot = torch.split(q_states, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
modeling_mistral_large4.py · L412
YaRN RoPE scaling code only in its codediscussed

rope_parameters.rope_type is 'yarn' and the rotary embedding module dispatches to the YaRN init function via ROPE_INIT_FUNCTIONS, with…

show evidence

rope_parameters.rope_type is 'yarn' and the rotary embedding module dispatches to the YaRN init function via ROPE_INIT_FUNCTIONS, with mscale applied to attention scaling.

text_config.rope_parameters = {"attention_factor": 1.0, "beta_fast": 32, "beta_slow": 1, "factor": 2048, "o…
self.rope_type = self.config.rope_parameters["rope_type"]
        rope_init_fn: Callable = self.compute_default_rope_parameters
        if self.rope_type != "default":
            rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
modeling_mistral_large4.py · L84–87
Layers without positional encoding (NoPE) code only in its codenot stated

no_rope_layers marks every 4th softmax-attention layer (value 0) to skip RoPE entirely while other layers apply rotary embeddings, all…

show evidence

no_rope_layers marks every 4th softmax-attention layer (value 0) to skip RoPE entirely while other layers apply rotary embeddings, all within the same indexed_attention layer type.

text_config.no_rope_layers = [1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1…
self.use_rope = config.no_rope_layers[layer_idx]
modeling_mistral_large4.py · L209
if self.use_rope:
            cos, sin = position_embeddings
            # For layers with cache sharing, only rotate the queries: the keys were already rotated by the source layer
            if self.is_kv_shared_layer:
                q_rot, _ = apply_rotary_pos_emb_interleave(q_rot, None, cos, sin)
modeling_mistral_large4.py · L425–429
Mixture of experts in its codeadopted

MistralLarge4TextMoE routes tokens to n_routed_experts=128 experts via a top-k router selecting num_experts_per_tok=4 experts per token…

show evidence

MistralLarge4TextMoE routes tokens to n_routed_experts=128 experts via a top-k router selecting num_experts_per_tok=4 experts per token, applied in all 'sparse' MLP layers.

text_config.n_routed_experts = 128text_config.num_experts_per_tok = 4
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_mistral_large4.py · L543–544
Shared expert in its codeadopted

MistralLarge4TextMoE adds a shared_experts MLP sized by moe_intermediate_size * n_shared_experts=1 that every token passes through in…

show evidence

MistralLarge4TextMoE adds a shared_experts MLP sized by moe_intermediate_size * n_shared_experts=1 that every token passes through in addition to routed experts.

text_config.n_shared_experts = 1
self.shared_experts = MistralLarge4TextMLP(
            config=config, intermediate_size=config.moe_intermediate_size * config.n_shared_experts
        )
modeling_mistral_large4.py · L601–603
hidden_states = hidden_states + self.shared_experts(residuals)
modeling_mistral_large4.py · L611
Auxiliary-loss-free load balancing (selection bias) not in its codenot stated

The router (MistralLarge4TextTopkRouter) computes softmax scores and selects top-k directly with group masking but applies no learned…

show evidence

The router (MistralLarge4TextTopkRouter) computes softmax scores and selects top-k directly with group masking but applies no learned per-expert selection bias (no e_score_correction_bias in config or code).

scores_for_choice = scores.masked_fill(~score_mask.bool(), 0.0)
        topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
modeling_mistral_large4.py · L542–543
Multi-token prediction layers not in its codenot stated

The code explicitly skips MTP weights ('mtp_dspark' keys ignored on load) and the config has no num_nextn_predict_layers/mtp_num_layers…

show evidence

The code explicitly skips MTP weights ('mtp_dspark' keys ignored on load) and the config has no num_nextn_predict_layers/mtp_num_layers count declaring MTP modules to build.

_keys_to_ignore_on_load_unexpected = [r"mtp_dspark\..*"]
modeling_mistral_large4.py · L684
SwiGLU feed-forward code only in its codenot stated

Both the dense MLP and MoE experts compute silu(gate_proj(x)) * up_proj(x), i.e. SwiGLU, with hidden_act set to silu.

show evidencetext_config.hidden_act = "silu"
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj
modeling_mistral_large4.py · L508–509

token mixer 10

softmax attention 2

SSMax used

“LeRoPE and SSMax let the model learn rotary frequencies and per-head query scaling.”

multi-head latent attention 1

Multi-head Latent Attention core

“Its attention blocks combine Multi-head Latent Attention (MLA) with a DeepSeek Sparse Attention (DSA) indexer”

sparse attention 8

Sparse attention core

“Mistral Large 4 combines a large pool of experts with sparse attention and a Pixtral vision encoder.”

FlashMLA sparse attention default

“FlashMLA sparse attention and an FP8 latent KV cache (fp8_ds_mla) by default”

DeepSeek Sparse Attention used

“we adapted ML4 to DSA, using lightweight indexers to select the top-2,048 key tokens for each query”

sparse attention indexer 5

DeepSeek Sparse Attention indexer core

“MLA keeps a compressed KV representation, and DSA selects up to 2,048 key positions for each query.”

Multi-head latent attention with a DSA sparse indexer core

“MLA attention with a DSA sparse indexer (top-2048 keys)”

KL-distillation indexer warm-up used

“we warm up the indexers for approximately 15.1B tokens through KL-based distillation of the main model’s attention distributions”

Shared DSA indexer across sliding-window layers used

“We therefore share a single DSA indexer among the former SWA layers within each of these groups.”

Top-2,048 key-token selection per query used

“using lightweight indexers to select the top-2,048 key tokens for each query”

channel mixer 7

mixture of experts 7

Mixture of experts core

“Open-weight hybrid instruct-and-reasoning MoE with multimodal input”

Hybrid instruction-and-reasoning MoE core

“Open-weight hybrid instruct-and-reasoning MoE with multimodal input”

expert routing 1

Top-k expert routing core

“a softmax router selects four of the 128 experts for each token.”

expert load balancing 1

Expert-parallel-group load-balancing loss used

“we compute the load-balancing loss over each expert-parallel (EP) group”

shared experts 1

Shared expert core

“128 routed experts (top-4) plus one shared expert”

fine-grained experts 2

Granular Mixture-of-Experts with routed and shared experts core

“granular Mixture-of-Experts with 128 routed experts (4 active per token) and 1 shared expert”

MoE granularity 4 used

“uses a granularity of 4 for its mixture-of-experts layers”

positional encoding 2

LeRoPE core

“Mistral uses LeRope, a YaRN-scaled RoPE with one learned multiplier per frequency.”

Alternating RoPE/NoPE layers core

“For each decoder block, there’s a 3:1 RoPE to NoPE pattern, i.e. 3 decoder layers employ RoPE and every fourth layer uses NoPE.”

multimodal architecture 5

Native multimodal architecture core

“our flagship 1-trillion-parameter, natively multimodal model”

Multimodal input core

“with multimodal input”

Pixtral vision encoder core

“Mistral Large 4 employs a 1.6B Pixtral encoder.”

Pixtral-style variable-resolution vision encoder core

“Pixtral-style vision encoder trained to handle variable image sizes and aspect ratios”

Spatial patch-merge projector core

“It uses 14 × 14 patches and a patch-merge projector with spatial merge size 2.”

2 more mentioned or evaluated
YaRN evaluated

positional encoding “we removed the attention windows and used YaRN to rescale the RoPE frequencies by 8×”

Load balancing mentioned

expert load balancing “Learn more about our load balancing and provider routing options.”

Not used 3

Cross-layer attention sharing not used

token mixer “Every other full-attention layer reuses the KV cache of the preceding full-attention layer.”

Sliding Window Attention not used

sliding window attention “the remaining layers use sliding-window attention (SWA) with a window size of 512 tokens”

Ragged Multi-head Latent Attention not used

multi-head latent attention “One in every four layers uses full attention, the remaining layers use sliding-window attention (SWA) with a window size of 512 tokens.”

distillation objective 1

Full-vocabulary logit distillation core

“we optimize the following per-token objective over the full vocabulary”

auxiliary loss 1

Reference-policy KL penalty using the k2 estimator optional

“For a subset of data sources prone to over-optimization, we add a KL penalty to the reference policy using the k 2 estimator [40] with…”

optimizer 3

Muon used

“We use Muon [38, 39] (Nesterov momentum 0.95, quintic Newton–Schulz with 10 iterations, spectral scaling) at a constant learning rate of…”

AdamW used

“We use the AdamW optimizer with (β1,β 2) = (0.975,0.975) during pretraining.”

Muon with Nesterov momentum and spectral scaling used

“(Nesterov momentum 0.95, quintic Newton–Schulz with 10 iterations, spectral scaling)”

learning-rate schedule 1

Batch-size warmup used

“we ramp up the batch size in stages with a share of tokens that grows exponentially per stage”

training precision 3

Blockwise scales in generator and trainer used

“and blockwise scales in both the generator and trainer.”

FP8 tensor-core training used

“we use FP8 tensor cores for fine-grained low-precision training”

NVFP4 optional

“A quantized NVFP4 checkpoint is also available, with a smaller memory footprint, faster inference and same performance.”

training stability 1

Router replay used

“router replay re-executes the expert-routing decisions made at inference time during the training forward pass, eliminating divergence…”

training parallelism 6

Expert parallelism used

“with expert parallelism, +14–17% prefill req/s at 32k”

Context parallelism used

“To mitigate this, we make use of context parallelism (CP) [80] to shard and de-replicate the KV cache across the sequence dimension instead.”

Communication-computation overlap used

“maximized compute–communication overlapping”

Latency-optimized context-parallel collectives used

“By rewriting the CP collectives with latency-optimized exchange patterns, in-kernel overlap of compute and communication, and other fusion…”

ZeRO-2 used

“We accelerate training by leveraging ZeRO-2 optimizations”

Prefill context parallelism optional

“Prefill context parallelism splits each prompt across the 8 ranks”

training runtime 5

Activation CPU offloading used

“the NVLink-C2C link to the Grace CPU for fast activation CPU offloading”

Asynchronous execution of supporting training workloads used

“asynchronous workloads to minimize impact on actual training (data loading, checkpointing, metrics, etc.)”

Automated health checks and crash analysis for training operations used

“automated health checks and crash analysis help identify failures and coordinate interventions across time zones.”

Fine-grained activation recomputation used

“fine-grained activation recomputation”

Fused full-logit KL-loss backward kernel used

“We implement a custom fused kernel for fast full-logit KL loss backward”

data filtering 1

Quality checks for generated instances used

“we perform a comprehensive set of checks, including hackability, problem–verifier mismatch, and insufficient learning signal”

synthetic data 7

Agent-based synthetic-data generation pipeline core

“we invested heavily in a high-quality, scalable agent-based pipeline to generate synthetic data”

Minimal-tool autonomous task authoring used

“We provide the authoring agent with a minimal set of tools to perform all necessary procedures to create realistic, high-quality tasks”

Refine software tasks using multiple solvers and trajectory inspection used

“A refinement agent runs several solvers of varying capabilities against each task and inspects their trajectories”

Reuse successful authoring-agent trajectories as midtraining data used

“We also include the successful authoring agent trajectories from these pipelines in our midtraining corpus”

Seed terminal-agent tasks with technical content from pretraining data used

“we collect millions of technical seeds from our pretraining corpus”

Taxonomy-guided generation of simulated workplace environments used

“By adopting taxonomies, we have fine-grained control over the coverage and complexity of the generated synthetic data.”

Two-stage GitHub-derived software task generation and RL refinement used

“We use a two-stage approach: first, an authoring agent turns each candidate into a runnable environment”

data mixture & curriculum 2

Heterogeneous training mixture core

“We optimize a highly heterogeneous training mixture covering multimodal, alignment, instruction-following, cybersecurity, code agents…”

Curriculum-driven data loading used

“The pipeline starts with curriculum-driven data loading”

sequence packing 1

Sequence packing used

“packing=True,”

tokenization 1

Trie-based tokenization used

“applies trie-based tokenization to handle possible re-tokenization events”

1 more mentioned or evaluated
Internal offline web mentioned

data sourcing “We plan to move towards an internal, offline version of the web to improve control over results and mitigate safety risks.”

supervised fine-tuning 6

Supervised fine-tuning used

“the final model owes them to both post-training stages (supervised fine-tuning and RL)”

Low-Rank Adaptation used

“peft_config=LoraConfig(”

Assistant-only loss used

“assistant_only_loss=True,”

Freeze the vision encoder during pretraining and unfreeze for SFT used

“frozen throughout pretraining and unfrozen only for supervised fine-tuning (SFT)”

Fine-tuning optional

“The model is free to use, modify, fine-tune, and commercialize”

Fine-tuning on traces from Mistral Large 4 optional

“with TRL you could fine-tune smaller models from Mistral Large 4 traces patching your code base.”

reinforcement learning algorithm 14

CISPO-style REINFORCE objective core

“We use a CISPO-style [34] REINFORCE [35] loss.”

Importance weighting for asynchronous policy lag core

“The objective corrects for this policy lag and numerical mismatch between the inference engine and the trainer by importance weighting…”

Keep importance-weighted tokens in the objective core

“Unlike PPO [36], the ratio is not clipped inside the objective, so tokens still contribute to learning when their weights are clipped.”

Per-token weight normalization by rollout group core

“Within a group, every trainable token gets the same weight, scaled so that the group’s total weight equals its number of rollouts”

Stop-gradient upper-clipped importance weighting core

“clipped from above at 3 (with no lower bound) and treated as a constant via a stop-gradient.”

Reinforcement learning used

“we've put that feedback to work while continuing to scale reinforcement learning, driving massive improvements in coding and defensive…”

Dropping zero-advantage rollout groups used

“When every rollout in a group gets the same reward, all its advantages are zero and it carries no learning signal. We drop such groups…”

Group-relative advantage estimation used

“A rollout’s advantage is its reward minus the group mean, and advantages are standardized across the batch.”

Group-based rollout sampling used

“For each prompt, we sample a group of rollouts. Group size is set per data source: 16 by default, 32 for agentic coding, and between 2 and…”

Group-centered batch-wide advantage normalization used

“We first subtract the groupwise mean reward, then normalize the resulting centered rewards by their standard deviation across the entire…”

MiMo-style segment-level advantage correction used

“For multi-turn trajectories, we additionally apply MiMo-style segment-level advantage correction [37] to turns flagged for generation…”

Off-policy drift reduction used

“Novel methods and optimizations across both stages minimize off-policy drift and enable stable RL over long horizons.”

Reinforcement-learning pipeline used

“Our RL pipeline enabled us to reach 84.5% on BrowseComp”

SFT checkpoint initialization for RL used

“starting from an SFT checkpoint that was achieving 75%”

reward modelling 12

Per-source composite reward functions core

“Each data source defines its own reward function by combining signals into a composite reward.”

Conservative failure attribution default

“Ambiguous cases default to the agent’s fault, so unreliable behavior is never silently dropped and implicitly rewarded.”

Adversarial-agent reward-hacking audit used

“We also employ an adversarial agent to check for reward-hacking vulnerabilities”

Composable multi-method reward verification used

“with reward models, unit tests, LLM judges, and static checks combined as needed for each task”

Exploit-neutralization and regression-free patch reward used

“A candidate patch receives positive reward only if it completely neutralizes the exploit while preserving existing test-suite passes.”

Fail-to-pass test verifier construction used

“Tests that fail before the reference solution and pass afterwards become the verifier”

Group-relative length bonus used

“we augment the task reward with a group-relative length bonus for correct rollouts.”

Pairwise rewards with groupwise penalties used

“which applies pairwise rewards and groupwise penalties”

Quantized group-relative rollout-length bonus used

“we quantize rollout lengths into bins of width ∆ = 0.1Lmin, anchored at Lmin.”

Reward-scale calibration used

“we design task rewards to have broadly comparable scales, with values typically in [0,1].”

Scientific-tool answer cross-checking used

“checks its answers against a standard chemistry tool.”

Reinforcement Learning with Verifiable Rewards optional

“reproduce-then-patch tasks use a verifiable reward, so it fits with RL environments that could be used to further RL smaller security…”

policy distillation 4

Multi-Teacher On-Policy Distillation used

“We combine these teachers into a single model through multi-teacher on-policy distillation [45, 46].”

Multiple-Teacher Distillation used

“Thanks to multiple-teacher distillation from checkpoints with different levels of verbosity”

Remote Teacher Infrastructure used

“We implement a remote teacher infrastructure so teachers can scale across separate GPU nodes, decoupling the number of teachers from…”

Three-Teacher Distillation for Reasoning-Effort Modes used

“For the final release model, we initialize the student from the high-reasoning RL checkpoint and train it with three teachers, yielding a…”

rollout & RL infrastructure 19

Asynchronous reinforcement learning core

“We use a custom, fully asynchronous RL training pipeline implemented in Ray”

Shared environment implementation across RL, evaluation, and synthetic data generation core

“All of our environments, from single-turn question answering (QA) to long-horizon agentic coding, are defined against a single interface.”

Adaptive Rollout Concurrency used

“The episode generation concurrency adapts following backpressure signals from generators, services, and the trainer, which limits staleness.”

Actor-pool orchestration for asynchronous RL used

“Our pipeline feeds RL instances through a series of actor pools that handle agentic-loop rollout, verification, scoring, and tokenization.”

Autoscaling rollout actors used

“an autoscaling fleet of actors generates tens of thousands of rollouts in parallel while model training proceeds asynchronously”

Autoscaling supporting services used

“All supporting services autoscale to adapt to the load imposed by in-progress rollouts.”

Backpressure- and concurrency-controlled rollout admission used

“admission of groups in the runner pool depending on maximum concurrency and backpressure signals”

Centralized trajectory bin-packing used

“A centralized bin-packer then packs individual trajectories into fixed-length microbatches sent to each data-parallel rank.”

Discarding infrastructure-failure rollouts in RL used

“rollout failures due to a sandbox crash or service call error are discarded in RL and retried in evaluation.”

Excluding auxiliary environment-model calls from training used

“Auxiliary model calls made inside the environment, such as user models and feedback digesters, are marked to avoid training on them. Only…”

Fixed-usable-trajectory-count batching used

“The trainer takes an optimizer step once it has accumulated a fixed number of usable trajectories (7k for ML4)”

Low-staleness optimization for long-trajectory RL used

“supporting rollout budgets of millions of tokens across multiple compactions while keeping staleness low”

Pause and resume in-flight rollouts across policy updates used

“In-flight rollouts pause, load the new weights, and resume without recomputing their KV cache.”

Per-step NCCL weight updates used

“Generators are served on Ray through vLLM engines [33] and receive weight updates via NCCL at each training step.”

Prioritizing sibling rollouts from started groups used

“with higher priority for sibling rollouts of started groups”

Provider failover routing used

“we can recover by routing to another healthy provider”

Remote teacher hidden-state transfer with trainer-side logit reconstruction used

“We reduce the communication cost between teachers and trainers by transferring only hidden states via NVIDIA Inference Xfer Library (NIXL)…”

Shared composable interface for RL tasks used

“A shared, composable interface allows a single training run to combine tasks ranging from single-turn chat and complex scientific problem…”

Submission-only termination for environments with a submit tool optional

“In environments with a submit tool, the per-turn verification is skipped and only submission ends the episode.”

agentic post-training 2

Common agent-environment interaction loop core

“Agentic tasks share a common loop: the agent acts until it produces a final message without tool calls. A verifier then scores the state…”

Terminal-task difficulty calibration using solver pass rates used

“use solver pass rates to calibrate the difficulty of each instance”

decoding strategy 8

DSpark core

“Both checkpoints ship the DSpark drafter. The speculative depth is fixed at 3 by the checkpoint.”

Three-token speculative draft depth core

“propose a block of 3 tokens”

Recurrent Markov draft head core

“with a recurrent Markov head”

Three-layer dense MLA speculative drafter core

“3 dense MLA draft layers in BF16 that propose a block of 3 tokens”

Temperature and nucleus sampling configuration used

“for all reasoning efforts, we recommend the following sampling parameters: temperature=1.0 top_p=0.95”

Block verification used

“Together with block verification [84], DSpark improves latency and throughput by 90% compared to a baseline without speculative decoding”

Recurrent conditioning head used

“we use a recurrent conditioning head and unique mask tokens [83], which improve acceptance length by +1.2% and +0.5%, respectively”

Unique mask tokens used

“we use a recurrent conditioning head and unique mask tokens [83], which improve acceptance length by +1.2% and +0.5%, respectively”

reasoning control 2

Thinking mode default

“three reasoning efforts — low ,medium ,high (default)”

Reasoning cannot be disabled default

“reasoning cannot be disabled, so using reasoning_effort: none would still use high.”

KV cache management 4

Cross-layer KV-cache sharing core

“cross-layer KV sharing”

Prefix caching default

“Prefix caching (on by default) for agentic workloads”

You Only Cache Once (YOCO) used

“Full-attention layers in the second half of the transformer attend to the KV cache produced by the first half instead of maintaining their…”

Decode context parallelism optional

“Decode context parallelism shards the latent KV cache across the 8 ranks”

inference quantization 8

FP8 latent KV-cache quantization default

“the latent KV cache is plain FP8 (E4M3) by default, 1.86x the capacity of bf16”

FP8 used

“we adopt FP8 inference to improve generation speed and match the trainer”

Mixed-precision NVFP4 routed-expert quantization used

“We also quantize ML4’s routed experts into NVFP4 format while keeping other components at their training precision, resulting in a mixed…”

Abs-max quantization used

“We found that naive abs-max quantization already suffices and matches the FP8 reference in all evaluations with appropriate calibration.”

Native FP8 KV cache with offloading used

“In combination with a native FP8 KV cache and offloading, this gives us approximately 20 million tokens of in-memory KV-cache space on…”

OCP MXFP4 quantization used

“The routed experts run as OCP MXFP4 on AITER's fused MoE”

SFT-mixture calibration used

“We use 200M tokens from our SFT mixture for calibration.”

Layer-by-layer FP8-to-MXFP4 expert re-quantization at load optional

“from the FP8 checkpoint re-quantized at load (VLLM_FP8_MOE_REQUANT=mxfp4, layer by layer, so peak load memory stays low)”

serving parallelism 2

Disaggregated serving across 4+4 GPUs used

“Disaggregated 4+4 beats colocated 8-GPU serving by 1.8–2.2x at 8k and 4–5x at 32k/60k”

Tensor and expert parallelism used

“The recipe runs vLLM with the FP8 weights split across all eight GPUs using tensor and expert parallelism”

inference scheduling 2

Metric-driven inference-fleet autoscaling used

“We autoscale our inference fleet based on live request and server metrics.”

Session-sticky routing to data-parallel ranks used

“Route each session to the same DP rank on DP sides”

inference kernel 5

Breakable CUDA graphs default

“vLLM serves the model on model runner V2 (the default) with breakable CUDA graphs”

Kernel fusion used

“systematic kernel fusing”

FA4 blockwise attention used

“We handle this setting correctly while maintaining efficiency by leveraging blockwise attention from FA4 [41].”

Persistent FlyDSL fused MoE decode kernel optional

“Runs every MoE layer of a 1-, 2- or 4-request decode step as one persistent FlyDSL kernel”

Single-launch fusion of attention, indexer, routing, experts, and all-reduces optional

“attention, indexer, router, MXFP4 experts and both all-reduces in one launch”

agentic scaffolding 5

Strict JSON Schema output core

“Mistral Large 4 enforces response formats natively, including strict JSON schemas.”

MicroVM-based sandbox isolation used

“We are now rebuilding our sandbox infrastructure around microVMs to strengthen isolation and support efficient snapshotting.”

Minimal tool-set design used

“keep the set of tools well-defined and limit their number to the minimum required for the use case”

System-prompt guidance for tool use used

“define a clear environment and use case, including guidance on how to effectively leverage tools in agentic systems”

Turn-end hook used

“Environments use this hook for multi-turn interaction: for example, feeding back a digest of failing tests as a hint, moving the agent on…”

1 more mentioned or evaluated
Provider routing mentioned

inference scheduling “Learn more about our load balancing and provider routing options.”

inference engine 2

llm-d routing with Model Express startup used

“We handle inference at scale by building on llm-d [86, 87] for routing and Model Express [88] for faster startup.”

TensorRT-LLM optional

“the open source TensorRT LLM library for high-throughput serving.”

training framework 1

TRL used

“We also ship day 0 support of fine tuning Mistral Large 4 using TRL.”

kernel & quantization library 3

AITER gfx950 DSA indexer kernels default

“the DSA indexer on AITER's gfx950 indexer kernels”

AITER sparse MLA decode kernels default

“The sparse MLA runs on AITER's MLA decode kernels (ROCM_AITER_MLA_SPARSE)”

MegaMoE FlashInfer expert-parallel CUTEDSL backend optional

“The preset adds EP, as used by the PCP prefill side of the P/D presets”

infrastructure service 4

Central scheduler used

“Our architecture is multi-cluster by design, and leverages a central scheduler with the option to spill over to cloud providers when…”

Model Express used

“we use Model Express for P2P checkpoint transfer over InfiniBand and warm up cache sharing.”

NIXL toy proxy used

“Routed with the NIXL toy proxy (tests/v1/kv_connector/nixl_integration/toy_proxy_server.py), not vllm-router.”

Ray data plane used

“We use the Ray data plane to transport heavy objects across the pipeline, such as MoE routing decisions for router replay.”

cybersecurity - vulnerability discovery; exploit development 2

Cybersecurity guardrails core

“Our model’s built-in guardrails distinguishes between defensive and malicious, offensive cybersecurity tasks.”

Exploit and functional-correctness testing used

“We automatically construct rigorous test suites that probe the affected paths using both adversarial exploits and standard inputs”

evaluation harness 1

Vulnerability test harness validation against unpatched and fixed code used

“A test harness is validated only when it triggers on the unpatched codebase and passes following the reference fix.”

human & real-world evaluation 1

Browser-based testing and iterative repair used

“uses it in a browser to catch its own bugs and slowdowns, then fixes what an independent reviewer flags until all requirements pass.”

filed at the root 3

Append-only episode records core

“Every request or response is logged to an append-only list of records, which is the single representation of an episode, from which…”

KV-producer weight initialization by copying used

“we added KV-producer weights by copying them from the last long layer that maintained its own KV weights”

Model surgery to remove KV-cache compression used

“The goal was to remove KV-cache compression mechanisms to achieve an architecture as close as possible to a full-attention model.”

unfiled 14

Customization with proprietary cybersecurity data and policies used

“customers are tailoring Mistral Large 4 with proprietary security data, policies, detection logic, and threat environments using Mistral…”

Episode replay to detect behavior changes after refactors used

“The records also support replay, which lets us check that refactors do not change environment behavior.”

Hardware multicast and latency-first fused context-parallel communication kernels used

“Improvements primarily come from hardware multicast, latency-first data-exchange patterns, removing in-kernel synchronization, and fusing…”

Monitor RL runs for reward hacking and environment leaks used

“We actively monitor our RL training runs for reward hacking and environment leaks.”

Multi-agent quality inspection of generated workplace environments used

“a multi-agent setup inspects each workspace for realism, consistency, and file integrity, before providing integrated feedback”

Online RL learning-signal and pipeline-efficiency monitoring used

“We monitor the quality of the learning signal and efficiency of the online training pipeline.”

Overlap teacher forward computation with the trainer's final forward batch used

“we overlap the teacher forward pass with the final batch of the trainer forward pass.”

Customization using proprietary organizational data optional

“Organizations can customize Mistral Large 4 around their proprietary data with Mistral Forge”

Cybersecurity-specialized model variant optional

“preview access to a specialized variant of Mistral Large 4 optimized for advanced cybersecurity workflows”

Expert-customer collaboration to build and deploy tailored AI systems optional

“Mistral’s Applied AI experts can work alongside customer teams to build and deploy these tailored systems.”

Managed-service deployment with private Azure deployment options optional

“For organizations that prefer a managed service, Mistral Large 4 is available through Microsoft Foundry, with private deployment options…”

Model customization optional

“Developers can use the open source NeMo AutoModel library for customization”

Self-hosted model deployment with inspectable, customizable weights optional

“Organizations have the ability to inspect, customize, and deploy the model on their own infrastructure.”

Task-specific model distillation or optimization into smaller variants optional

“For well-defined cybersecurity tasks, it can be further optimized into smaller, task-specific variants for greater accuracy”

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

SourceKindPublisherStatus
mistral.ai/news/mistral-large-4-ossofficial blogMistral AIread
mistral.ai/news/mistral-large-4official blogMistral AIread
huggingface.co/mistralai/Mistral-Large-4-1T-A52B/blob/main/assets/techn…technical reportMistral AIread
huggingface.co/mistralai/Mistral-Large-4-1T-A52Bmodel cardMistral AIread
huggingface.co/blog/mistral-large4third party analysisHugging Faceread
recipes.vllm.ai/mistralai/Mistral-Large-4-1T-A52Bvendor docsvLLMread
docs.mistral.ai/models/mistral-large-4-0vendor docsMistral AInot fetched
openrouter.ai/mistralai/mistral-large-4-0vendor docsOpenRouterread
↑