DeepSeek
DeepSeek-V4-Pro
What DeepSeek-V4-Pro’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 1.6T / 49B active
- Context
- 1M
- Attention
- sparse (indexer)Compressed Sparse Attention + Heavily Compressed Attention with indexer, sliding window 128, attention sink
- Experts
- 384 routed + 1 shared, top-6
- Input
- text
- Licence
- MIT
- Released
- 2026-04-24
Against the consensus recipe
Its documents state 7 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictioncore
- Grouped-query attentionnot in its documents
Optimization
- Muoncore
- Expert parallelismnot in its documents
- Quantization-Aware Trainingused
Post-training
- Supervised fine-tuningused
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationcore
Inference & serving
- Thinking modeused
- Multi-Token Predictionnot in its documents
- FP8not in its documents
Checked in its code
Its code shows 11 of the 19 architecture features checked — 4 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_deepseek_v4.py, configuration_deepseek_v4.py) and config.json at revision b5968e9190, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention | not in its code | not stated | num_key_value_heads=1 implements shared-KV multi-query attention, but the single KV vector is a low-rank-projected latent reused via… show evidencenum_key_value_heads=1 implements shared-KV multi-query attention, but the single KV vector is a low-rank-projected latent reused via broadcasting rather than grouped multi-head KV projection, and the model otherwise relies on MLA-style projections, so GQA as a head-grouping mechanism with multiple real KV heads is not present (it's MQA/MLA-like with one shared KV head). num_key_value_heads = 1num_attention_heads = 128 |
| Multi-head latent attention | not in its code | not stated | There is no kv_lora_rank; keys/values are not down-projected into a shared low-rank latent and up-projected per head, instead a single… show evidenceThere is no kv_lora_rank; keys/values are not down-projected into a shared low-rank latent and up-projected per head, instead a single shared KV head (head_dim=512) is produced directly by kv_proj, so this is not DeepSeek-V2-style MLA. q_lora_rank = 1536 |
| Sliding-window attention | in its code | adopted | Every attention layer (sliding_attention, CSA, HCA) uses a sliding-window cache of size config.sliding_window=128 as its base local K/V… show evidenceEvery attention layer (sliding_attention, CSA, HCA) uses a sliding-window cache of size config.sliding_window=128 as its base local K/V window. sliding_window = 128 |
| Interleaved sliding-window and global attention | code unclear | not stated | layer_types is derived at runtime from compress_ratios (ending with a 0 which maps to sliding_attention, others to… show evidencelayer_types is derived at runtime from compress_ratios (ending with a 0 which maps to sliding_attention, others to compressed_sparse_attention/heavily_compressed_attention), but all layer types still apply the same sliding_window base cache, and compress-attention layers additionally concatenate long-range compressed KV rather than attending to the genuinely full context, so whether any layer is truly 'full-context' versus all being sliding-window-based cannot be settled from the given files. sliding_window = 128 |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | in its code | adopted | DeepseekV4Indexer scores compressed KV entries with index_n_heads/index_head_dim and keeps only the top index_topk=1024 entries per query… show evidenceDeepseekV4Indexer scores compressed KV entries with index_n_heads/index_head_dim and keeps only the top index_topk=1024 entries per query for CSA layers. index_topk = 1024index_n_heads = 64index_head_dim = 128 |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | No linear-attention or state-space recurrent mixer layers exist; all layer_types (sliding_attention, compressed_sparse_attention… show the whole verdictNo linear-attention or state-space recurrent mixer layers exist; all layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) are softmax-attention variants implemented by DeepseekV4Attention. |
| Gated DeltaNet layers | not in its code | not stated | There is no gated delta-rule recurrent state update implemented anywhere in this file; all token mixing is softmax attention. |
| Mamba-2 layers | not in its code | not stated | No Mamba-2/SSD selective state-space block is implemented in this file. |
| Learnable attention sink code only | in its code | not stated | Each attention layer has a per-head learnable sink parameter concatenated into the softmax denominator in eager_attention_forward. show evidence |
| Gated attention output | not in its code | not stated | The attention output is not multiplied by an input-dependent sigmoid gate; instead it undergoes a conjugate RoPE rotation and a grouped… show evidenceThe attention output is not multiplied by an input-dependent sigmoid gate; instead it undergoes a conjugate RoPE rotation and a grouped output projection, with no sigmoid gating mechanism present. |
| QK normalization code only | in its code | not stated | Queries are normalized with an unweighted RMSNorm per head (q_b_norm) before the attention dot product; keys (the shared KV) are… show evidenceQueries are normalized with an unweighted RMSNorm per head (q_b_norm) before the attention dot product; keys (the shared KV) are normalized with kv_norm as well. |
| Partial RoPE code only | in its code | not stated | RoPE is applied only to the trailing qk_rope_head_dim=64 of each 512-dim head (partial_rotary_factor=64/512), leaving the rest of the head… show evidenceRoPE is applied only to the trailing qk_rope_head_dim=64 of each 512-dim head (partial_rotary_factor=64/512), leaving the rest of the head dimension without position signal. qk_rope_head_dim = 64head_dim = 512 |
| YaRN RoPE scaling code only | in its code | not stated | rope_scaling.type is yarn and is wired into the compress rope-type parameters that the YaRN-capable ROPE_INIT_FUNCTIONS dispatch consumes… show evidencerope_scaling.type is yarn and is wired into the compress rope-type parameters that the YaRN-capable ROPE_INIT_FUNCTIONS dispatch consumes for compressed-attention layers. rope_scaling = {"beta_fast": 32, "beta_slow": 1, "factor": 16, "original_max_position_embedd… |
| Layers without positional encoding (NoPE) | not in its code | not stated | All attention layers apply partial RoPE (either 'main' or 'compress' rope type) to their rope slice; there is no attention layer that… show evidenceAll attention layers apply partial RoPE (either 'main' or 'compress' rope type) to their rope slice; there is no attention layer that skips positional encoding entirely. |
| Mixture of experts | in its code | adopted | Feed-forward layers route tokens to n_routed_experts=384 expert MLPs via a learned TopKRouter/HashRouter selecting num_experts_per_tok=6… show evidenceFeed-forward layers route tokens to n_routed_experts=384 expert MLPs via a learned TopKRouter/HashRouter selecting num_experts_per_tok=6 experts. n_routed_experts = 384num_experts_per_tok = 6 |
| Shared expert | in its code | adopted | Every MoE layer adds one shared DeepseekV4MLP expert that all tokens pass through, with n_shared_experts=1. show evidencen_shared_experts = 1 |
| Auxiliary-loss-free load balancing (selection bias) | in its code | adopted | The router adds e_score_correction_bias to scores only for top-k selection (not for the weights used to scale expert outputs), matching… show evidenceThe router adds e_score_correction_bias to scores only for top-k selection (not for the weights used to scale expert outputs), matching DeepSeek-V3's bias-based auxiliary-loss-free balancing, enabled via topk_method noaux_tc. topk_method = "noaux_tc" |
| Multi-token prediction layers | in its code | adopted | The config declares num_nextn_predict_layers=1 (MTP module count), and the modeling file's ignore-on-load pattern for 'mtp.' prefixed keys… show evidenceThe config declares num_nextn_predict_layers=1 (MTP module count), and the modeling file's ignore-on-load pattern for 'mtp.' prefixed keys confirms the checkpoint carries these extra modules even though they are not built in this file. num_nextn_predict_layers = 1 |
| SwiGLU feed-forward | in its code | adopted | Both the dense DeepseekV4MLP and the MoE DeepseekV4Experts compute act_fn(gate)*up with hidden_act=silu, i.e. SwiGLU. show evidencehidden_act = "silu" |
token mixer 9
softmax attention 3
sliding window attention 1
“we introduce a supplementary attention branch to both CSA and HCA in a sliding window manner”
grouped-query attention 1
“CSA then performs core attention in a Multi-Query Attention (MQA) manner”
attention sink 1
“we employ the trick of attention sink”
sparse attention 5
“Novel Attention: Token-wise compression + DSA (DeepSeek Sparse Attention)”
“we design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)”
“we design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)”
“Novel Attention: Token-wise compression + DSA (DeepSeek Sparse Attention)”
“the strategy of introducing sparse attention is the same as DeepSeek-V4-Flash, following a two-stage training method.”
hybrid layer stacking 1
“Hybrid Attention Architecture: We design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed…”
channel mixer 10
dense feed-forward network 2
“each token-expert pair requires FLOPs (SwiGLU gate, up, and down projections)”
“applying SwiGLU clamping effectively eliminates outliers and substantially aids in stabilizing the training process”
mixture of experts 8
“two strong Mixture-of-Experts (MoE) language models — DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSeek-V4-Flash with 284B…”
“DeepSeek-V4 series retain the DeepSeekMoE framework”
“We have open-sourced the CUDA-based mega-kernel implementation named MegaMoE”
expert routing 3
“Anticipatory Routing (computing routing indices one step ahead to decouple backbone and routing updates)”
“we replace the dense FFN layers in the initial several Transformer blocks with MoE layers that employ Hash routing”
“we introduced an automatic detection mechanism that triggers a short rollback and activates Anticipatory Routing exclusively when a loss…”
expert load balancing 1
“For auxiliary-loss-free load balancing, we set the bias update speed to 0.001.”
positional encoding 1
“we partially employ the Rotary Positional Embedding (RoPE) to the attention queries, KV entries, and the core attention outputs”
normalization & residual 2
“DeepSeek-V4 series incorporate Manifold-Constrained Hyper-Connections (mHC) to strengthen the conventional residual connections”
“we perform an additional RMSNorm operation on each head of the queries and the only head of the compressed KV entries”
prediction head 1
“DeepSeek-V4 series also set MTP modules and objectives”
context capacity 1
“1M context is now the default across all official DeepSeek services”
distillation objective 1
“we adopt full-vocabulary logit distillation in our OPD.”
auxiliary loss 1
“we employ sample-level attention masking during pre-training.”
optimization 17
optimizer 4
“We employ the Muon optimizer for the majority of modules in DeepSeek-V4 series”
“We maintain the AdamW optimizer for the embedding module, the prediction head module, the static biases and gating factors of mHC modules…”
“we also apply weight decay to Muon parameters, use the Nesterov trick”
“we use hybrid Newton-Schulz iterations for orthogonalization”
learning-rate schedule 1
“we also employ a batch size scheduling strategy that increases the batch size (in tokens) from a small size to 75.5M”
training precision 1
“FP4 + FP8 Mixed: MoE expert parameters use FP4 precision; most other parameters use FP8.”
quantization-aware training 4
“we introduce Quantization-Aware Training (QAT) [QAT] during the post-training stage”
“we incorporate FP4 quantization-aware training for MoE expert weights and the indexer QK path”
“We apply FP4 (MXFP4) quantization [OCP_MXFormat] to two components”
“equivalent to applying the Straight-Through Estimator (STE) through the quantization operation”
training parallelism 4
“First, an all-to-all operation exchanges local gradients across ranks, and then each rank performs a local sum in FP32.”
“To address this conflict, we design a hybrid strategy of ZeRO bucket assignment for Muon.”
“we employ a knapsack algorithm to assign parameter matrices to these ranks, ensuring each rank manages a roughly balanced load.”
“we adjust the DualPipe 1F1B overlapping scheme to accommodate the increased pipeline communication”
training runtime 3
“we implement a tensor-level activation checkpointing mechanism with automatic differentiation support.”
“For each annotated tensor, it performs a backward traversal to identify the minimal subgraph required for its recomputation.”
“we introduce a recomputation strategy that selectively checkpoints intermediate tensors.”
filed at the root 1
“V4 also expands its multilingual corpus”
data sourcing 1
“V4 places particular emphasis on scientific papers, technical reports”
data filtering 1
“we implement filtering strategies to remove batched auto-generated and templated content”
synthetic data 2
“DeepSeek curates rubric-guided RL data”
“DeepSeek incorporates agentic data during what it calls the 'mid-training phase'”
sequence packing 1
“we pack documents from different sources into appropriate sequences to minimize sample truncation.”
tokenization 1
“We append a set of dedicated special tokens directly to the input sequence, where each token corresponds to a specific auxiliary task.”
supervised fine-tuning 2
“independent cultivation of domain-specific experts (through SFT and RL with GRPO)”
“DeepSeek trains independent specialist models — one for mathematics, one for coding, one for agent tasks, one for instruction following”
reinforcement learning algorithm 2
“followed by Reinforcement Learning using Group Relative Policy Optimization (GRPO)”
“For the RL stage, we implemented the Group Relative Policy Optimization (GRPO) algorithm”
reward modelling 2
“we curate rubric-guided RL data and employ a Generative Reward Model (GRM) to evaluate policy trajectories.”
“Crucially, we apply RL optimization directly to the GRM itself.”
policy distillation 2
“we employ multi-teacher On-Policy Distillation (OPD) as the primary technique for merging expert capabilities into the final model.”
“post-training pipeline replaces reinforcement learning with On-Policy Distillation from 10+ domain specialists”
rollout & RL infrastructure 1
“The preemptible rollout service preserves partial generations through a token-granular Write-Ahead Log.”
mid-training & continual pretraining 1
“DeepSeek incorporates agentic data during what it calls the 'mid-training phase'”
reasoning control 3
“Both V4-Pro and V4-Flash support three modes: Non-think, Think High, and Think Max”
“all reasoning content is fully preserved throughout the entire conversation.”
“we utilize specialized response formats demarcated by the <think> and </think> tokens.”
KV cache management 6
“we design a customized KV cache layout.”
“we pre-allocate a fixed- and limited-size pool of state caches, and dynamically assign it to each sequence.”
“Achieving this requires co-designing the KV cache layout and the sparse attention kernel.”
“we leverage an on-disk KV cache storage mechanism to eliminate repeated prefilling for shared-prefix requests.”
“This strategy checkpoints SWA KV entries of the last tokens within every tokens, where is a tunable parameter.”
“This strategy does not store any SWA KV entries.”
inference quantization 2
“attention computation within the lightning indexer is performed in FP4 precision”
“FP8 precision is applied to the remaining dimensions”
inference kernel 9
“we implement end-to-end, bitwise batch-invariant, and deterministic kernels with minimal performance overhead”
“Therefore, we replace it end-to-end with DeepGEMM.”
“the second kernel utilizes distributed shared memory within thread-block clusters, enabling high-speed data exchange across SMs”
“To address this, we develop a dual-kernel strategy for batch-invariant decoding.”
“we carefully design and implement fused kernels of mHC for both training and inference.”
“We mitigate this overhead with Host Codegen, which moves most host-side logic into generated host code”
“we allocate separate accumulation buffers for each SM, followed by a global deterministic summation across all buffers.”
“we output each split part separately and perform a deterministic reduction in a subsequent kernel”
“we design a token order pre-processing mechanism within each single rank, combined with buffer isolation across multiple ranks.”
agentic scaffolding 4
“we introduce a new tool-call schema that employs a special "|DSML|" token and utilizes an XML-based format for tool invocations”
“the "thinking" mode utilizes agentic search.”
“the "non-think" mode employs Retrieval-Augmented Search (RAG)”
“The sandbox infrastructure (DSec) manages hundreds of thousands of concurrent sandbox instances”
Not used 1
inference kernel “Therefore, we abandon split-k in most scenarios, which, however, may cause performance degradation.”
infrastructure service 2
“we implement a token-granular Write-Ahead Log (WAL) for each generation request.”
“we integrate the Z3 SMT solver into TileLang’s algebraic system”
human & real-world evaluation 3
“DeepSeek curated approximately 200 tasks from 50+ internal engineers across PyTorch, CUDA, Rust, and C++”
“DeepSeek ran pairwise comparisons across 3,170 functional writing tasks and 2,837 creative writing tasks.”
“DeepSeek built a suite of 30 advanced Chinese professional workflows spanning 13 industries”
unfiled 5
“we introduce a finer-grained expert partitioning scheme... forms a fine-grained pipeline among experts”
“comprehensive post-training pipeline. The post-training features a two-stage paradigm: independent cultivation of domain-specific experts…”
“We adopt TileLang to develop a set of fused kernels to replace the vast majority of them”
“To address these challenges, we design a two-stage communication approach.”
1 more mentioned or evaluated
“candidate natural-language solutions are first generated and filtered by…”
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- DeepSeek-V4-Pro architecture card — LLM Architecture Gallery explainer sebastianraschka.com
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| api-docs.deepseek.com/news/news260424 | official blog | DeepSeek | read |
| arxiv.org/abs/2606.19348 | technical report | DeepSeek | read |
| fe-static.deepseek.com/chat/transparency/deepseek-V4-model-card-EN.pdf | technical report | DeepSeek | read |
| huggingface.co/deepseek-ai/DeepSeek-V4-Pro | model card | DeepSeek | read |
| morphllm.com/deepseek-v4 | third party analysis | Morph | not fetched |
| kili-technology.com/blog/data-story-deepseek-v4 | third party analysis | Kili Technology | read |
| recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Pro | vendor docs | vLLM | not fetched |
| lmsys.org/blog/2026-04-25-deepseek-v4 | vendor docs | LMSYS | not fetched |
| docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4 | vendor docs | SGLang | not fetched |