Model techniques map
ModelsDeepSeek-V4-Pro

DeepSeek · released 2026-04-24

DeepSeek-V4-Pro

What DeepSeek-V4-Pro’s own documents say it is built from — every method with the sentence that describes it.

OpenRouterHugging Face

Curator’s note1.6T total / 49B active MoE, 1M context, >32T pretraining tokens. Shares the arXiv 2606.19348 report with V4-Flash — the extractor will see the same report twice; dedup by content_hash or accept the double mention.

Against the consensus recipe

Its documents state 9 of the 15 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something the documents do not say, not something the model lacks.

  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 12 of the 19 architecture features checked — 5 of them are not stated in its documents.

Documents say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_deepseek_v4.py, configuration_deepseek_v4.py) and config.json at revision b5968e9190, by anthropic/claude-sonnet-5 on 2026-09-25.

FeatureCodeIts documentsEvidence
Grouped-query attention code only in its codenot stated

num_key_value_heads is 1 (shared-KV multi-query attention) with 128 query heads all broadcasting from a single KV head via repeat_kv

num_key_value_heads = 1num_attention_heads = 128
self.num_heads = config.num_attention_heads
        self.num_key_value_groups = config.num_attention_heads  # single KV head, broadcast to all
        self.head_dim = config.head_dim
modeling_deepseek_v4.py · L781–783
Multi-head latent attention not in its codenot stated

Attention uses a single shared KV head with q_lora_rank down-projection for queries but no kv_lora_rank latent down/up-projection for keys/values as in DeepSeek-V2 MLA; kv_proj projects directly to one head_dim-sized KV, not a shared low-rank latent

q_lora_rank = 1536
self.kv_proj = nn.Linear(config.hidden_size, self.head_dim, bias=False)
        self.kv_norm = DeepseekV4RMSNorm(self.head_dim, eps=config.rms_norm_eps)
modeling_deepseek_v4.py · L793–794
Sliding-window attention in its code used

sliding_window is 128 and every attention layer type (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) uses a sliding-window K/V cache with this window size

sliding_window = 128
self.cumulative_length += key_states.shape[-2]
        full = torch.cat([self.keys, key_states], dim=-2)
        self.keys = full[:, :, -self.sliding_window + 1 :, :]
modeling_deepseek_v4.py · L203–205
Interleaved sliding-window and global attention code unclearnot stated

layer_types cycles through sliding_attention/compressed_sparse_attention/heavily_compressed_attention but all three branches apply the same sliding_window cache underneath (CSA/HCA add extra compressed long-range KV on top), so whether any layer truly attends to full context is not settled by the given files, and layer_types itself is not explicitly given in this config (derived by default logic from compress_ratios)

compress_ratios = [128, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4,…
Indexer-selected sparse attention (DeepSeek Sparse Attention) in its code core

The CSA compressor runs a Lightning Indexer that scores compressed KV blocks and each query attends only to the top index_topk=1024 selected blocks

index_topk = 1024index_n_heads = 64index_head_dim = 128
top_k_indices = index_scores.topk(top_k, dim=-1).indices  # [B, S, k]
            invalid = top_k_indices >= causal_threshold.unsqueeze(-1)
            return torch.where(invalid, torch.full_like(top_k_indices, -1), top_k_indices)
modeling_deepseek_v4.py · L585–587
Linear-attention or state-space layers alongside full attention not in its codenot stated

No linear-attention, Gated DeltaNet, KDA, or Mamba token-mixing layers exist in this file; all layer_types variants are softmax attention with sliding-window plus compressor branches

Gated DeltaNet layers not in its codenot stated

No gated delta-rule recurrent mixer implementation is present in the given files

Mamba-2 layers not in its codenot stated

No Mamba-2 SSM blocks are implemented in the given files

Learnable attention sink code only in its codenot stated

Each attention layer has a per-head learnable sink parameter concatenated into the softmax denominator

sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_deepseek_v4.py · L736–737
Gated attention output not in its codenot stated

No sigmoid gate is applied to the attention output before the output projection; the attention output goes through rope-conjugate rotation and the grouped output projection directly

QK normalization code only in its codenot stated

Queries and keys (kv) are each passed through an RMSNorm (q_b_norm, kv_norm) before rotary application and the attention dot product

q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)

        kv = self.kv_norm(self.kv_proj(hidden_states)).view(*hidden_shape).transpose(1, 2)
modeling_deepseek_v4.py · L820–824
Partial RoPE code only in its codenot stated

RoPE is applied only to the trailing qk_rope_head_dim=64 slice of each 512-dim head, leaving the leading nope portion unrotated

qk_rope_head_dim = 64head_dim = 512
rope_dim = cos.shape[-1]
    nope, rope = x[..., :-rope_dim], x[..., -rope_dim:]
    rotated = ((rope.float() * cos) + (rotate_half(rope).float() * sin)).to(x.dtype)
    return torch.cat([nope, rotated], dim=-1)
modeling_deepseek_v4.py · L359–362
YaRN RoPE scaling code only in its codenot stated

rope_scaling type is yarn and is applied to the compress rope-type branch used by CSA/HCA layers via ROPE_INIT_FUNCTIONS

rope_scaling = {"beta_fast": 32, "beta_slow": 1, "factor": 16, "original_max_position_embedd…
compress.setdefault("rope_type", "default")
            if compress["rope_type"] == "yarn":
                compress.setdefault("attention_factor", 1.0)
configuration_deepseek_v4.py · L318–320
Layers without positional encoding (NoPE) not in its codenot stated

Every attention layer applies partial RoPE to its rope slice (sliding layers use 'main' rope, CSA/HCA layers use 'compress' rope); no layer type skips positional encoding entirely for its softmax attention

self.rope_layer_type = "main" if self.layer_type == "sliding_attention" else "compress"
modeling_deepseek_v4.py · L780
Mixture of experts in its code core

Feed-forward MoE layers route tokens to top-6 of 384 routed experts via a learned router

n_routed_experts = 384num_experts_per_tok = 6
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1086–1087
Shared expert in its code core

n_shared_experts is 1 and every MoE block adds a DeepseekV4MLP shared expert output to the routed output for every token

n_shared_experts = 1
routed = self.experts(flat, indices, weights).view(batch, seq_len, hidden_dim)
        return routed + self.shared_experts(residual)
modeling_deepseek_v4.py · L1139–1140
Auxiliary-loss-free load balancing (selection bias) in its code used

The router adds an e_score_correction_bias buffer to scores only for top-k expert selection, matching DeepSeek-V3's noaux_tc scheme, and topk_method is set to noaux_tc

topk_method = "noaux_tc"
self.e_score_correction_bias = nn.Buffer(torch.zeros(self.num_experts))

    def forward(self, hidden_states: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
        flat = hidden_states.reshape(-1, self.hidden_dim)
        logits = F.linear(flat, self.weight)
        scores = self.score_fn(logits)
        indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
modeling_deepseek_v4.py · L1080–1086
Multi-token prediction layers in its code core

num_nextn_predict_layers is 1 in this config, and the modeling code declares an ignore pattern for mtp weights, indicating declared MTP modules in the checkpoint

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_deepseek_v4.py · L1259
SwiGLU feed-forward in its code used

hidden_act is silu and both the dense MLP and the MoE experts compute act_fn(gate) * up with SiLU activation

hidden_act = "silu"
gate = self.gate_proj(x).clamp(max=self.limit)
        up = self.up_proj(x).clamp(min=-self.limit, max=self.limit)
        return self.down_proj(self.act_fn(gate) * up)
modeling_deepseek_v4.py · L1025–1027

model architecture 26

token mixer 9

softmax attention 3

sliding window attention 1

Sliding Window Attention used

grouped-query attention 1

Multi-query attention used

attention sink 1

Attention sink used

hybrid layer stacking 1

Hybrid Attention used

channel mixer 10

dense feed-forward network 2

SwiGLU used
SwiGLU clamping used

mixture of experts 8

Mixture of Experts core
DeepSeekMoE core
MegaMoE used

expert load balancing 1

Auxiliary-loss-free load balancing used

shared experts 1

Shared Experts core

positional encoding 1

Rotary Position Embedding used

normalization & residual 3

Manifold-Constrained Hyper-Connections core
RMSNorm used
QK-Clip not used

prediction head 1

Multi-Token Prediction core

context capacity 2

1M-token context window default
Engram not used

training objective 2

distillation objective 1

Full-vocabulary logit distillation used

auxiliary loss 1

Sample-level attention masking used

optimization 17

optimizer 4

Muon core
AdamW used
Nesterov momentum used
Hybrid Newton–Schulz iterations used

learning-rate schedule 1

Scheduled batch-size growth used

training precision 1

FP4+FP8 mixed precision used

quantization-aware training 4

Quantization-Aware Training used
FP4 Quantization used
FP4 Quantization-Aware Training used
Straight-Through Estimator used

training parallelism 4

All-to-all Gradient Exchange with Local FP32 Summation used
Hybrid ZeRO Bucket Assignment for Muon used
Knapsack-Based Balanced Assignment of Dense Parameter Matrices used
Modified DualPipe 1F1B Pipeline Overlap for mHC used

training runtime 3

Tensor-level automatic-differentiation activation checkpointing core
Minimal recomputation-graph extraction by backward traversal used
Selective tensor recomputation for activation-memory reduction used

data curation 7

filed at the root 1

Multilingual corpus expansion used

data sourcing 1

Long-document curation used

data filtering 1

Filtering batched auto-generated and templated content used

synthetic data 2

Rubric-guided RL data used
Task-specific data injection during mid-training used

sequence packing 1

Cross-source document packing used

tokenization 1

Quick Instruction tokens used

post-training 10

supervised fine-tuning 2

Supervised fine-tuning used
Specialist-model training used

reinforcement learning algorithm 2

Group Relative Policy Optimization used
Reinforcement Learning used

reward modelling 2

Generative Reward Model used
Direct RL optimization of a generative reward model used

policy distillation 2

Multi-Teacher On-Policy Distillation core
On-Policy Distillation core

rollout & RL infrastructure 1

Preemptible rollout service used

mid-training & continual pretraining 1

Mid-training phase used

inference & serving 26

reasoning control 4

Configurable reasoning effort used
Cross-turn persistent reasoning history used
Think-tag response formatting used
Thinking mode selection optional

KV cache management 6

Customized heterogeneous KV-cache layout core
Dynamic allocation of fixed-size state-cache pools used
KV-cache and sparse-attention-kernel co-design used
On-disk KV-cache storage used
Periodic cache checkpointing optional
Zero SWA caching optional

inference quantization 2

FP8 KV-cache quantization used
FP4 precision for the attention indexer used

inference kernel 10

Batch-invariant deterministic kernels core
DeepGEMM-based batch-invariant matrix multiplication used
Distributed shared memory for cross-SM attention data exchange used
Dual-kernel batch-invariant attention decoding used
Fused mHC kernels used
Host Codegen used
Per-SM accumulation buffers with deterministic global summation used
Separate split-K outputs with deterministic reduction used
Token-order preprocessing and cross-rank buffer isolation for deterministic MoE backward used
Avoiding split-K not used

agentic scaffolding 4

XML-based tool-call schema with DSML token used
Agentic search used
Retrieval-Augmented Search used
Sandbox infrastructure (DSec) used

software implementation 2

infrastructure service 2

Write-ahead log used
Z3 SMT solver integration used

evaluation 3

human & real-world evaluation 3

Internal engineering task evaluation used
Pairwise comparison for Chinese writing used
White-collar enterprise task evaluation used

unfiled 5

fine-grained EP scheme used
Post-training pipeline used
TileLang used
Two-stage contextual-parallel communication for compressed attention used
self-verification evaluated

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated document list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

DocumentKindPublisherStatus
api-docs.deepseek.com/news/news260424official blogDeepSeekread
arxiv.org/abs/2606.19348technical reportDeepSeekread
fe-static.deepseek.com/chat/transparency/deepseek-V4-model-card-EN.pdftechnical reportDeepSeekread
huggingface.co/deepseek-ai/DeepSeek-V4-Promodel cardDeepSeekread
morphllm.com/deepseek-v4third party analysisMorphnot fetched
kili-technology.com/blog/data-story-deepseek-v4third party analysisKili Technologyread