DeepSeek
DeepSeek-V4-Pro-0813
What DeepSeek-V4-Pro-0813’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 1.6T / 49B active
- Context
- 1M
- Attention
- sparse (indexer)Compressed Sparse Attention + Heavily Compressed Attention with indexer, sliding window 128, attention sink
- Experts
- 384 routed + 1 shared, top-6
- Input
- text
- Licence
- MIT
- Released
- 2026-08-13
Against the consensus recipe
Its documents state 1 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertsnot in its documents
- Multi-Token Predictionnot in its documents
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modeoptional
- Multi-Token Predictionnot in its documents
- FP8not in its documents
Checked in its code
Its code shows 12 of the 19 architecture features checked — 12 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_deepseek_v4.py, configuration_deepseek_v4.py) and config.json at revision 72e1d3230f, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention | not in its code | not stated | num_key_value_heads=1 with the single KV head broadcast to all query heads is multi-query attention implemented via repeat_kv, but the… show evidencenum_key_value_heads=1 with the single KV head broadcast to all query heads is multi-query attention implemented via repeat_kv, but the attention itself is MLA (latent KV projection) rather than classic GQA key/value projections, and the check explicitly excludes MLA-style attention num_key_value_heads = 1 |
| Multi-head latent attention code only | in its code | not stated | Queries and KV are down-projected through low-rank lora ranks (q_lora_rank, kv via kv_proj/kv_norm) and up-projected per head… show evidenceQueries and KV are down-projected through low-rank lora ranks (q_lora_rank, kv via kv_proj/kv_norm) and up-projected per head, DeepSeek-style, with a decoupled rope key applied via partial rotary q_lora_rank = 1536qk_rope_head_dim = 64 |
| Sliding-window attention code only | in its code | not stated | Every attention layer uses a sliding-window KV cache (DynamicSlidingWindowLayer) with sliding_window=128 passed to the attention interface show evidencesliding_window = 128 |
| Interleaved sliding-window and global attention | not in its code | not stated | All layers apply the same sliding-window K cache regardless of layer_types (sliding_attention, compressed_sparse_attention… show evidenceAll layers apply the same sliding-window K cache regardless of layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention all share the sliding-window branch); the compressor variants add extra long-range compressed KV on top, but none attend to the literal full context without a window, so there is no interleaving of sliding vs full/global attention sliding_window = 128 |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) code only | in its code | not stated | DeepseekV4Indexer scores compressed KV entries with ReLU(q·k) weighted sums and the CSA compressor selects the top index_topk=1024 entries… show evidenceDeepseekV4Indexer scores compressed KV entries with ReLU(q·k) weighted sums and the CSA compressor selects the top index_topk=1024 entries per query via index_scores.topk index_topk = 1024index_n_heads = 64index_head_dim = 128 |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | No recurrent linear-attention or SSM mixer class exists in this file; all three layer_types (sliding_attention… show the whole verdictNo recurrent linear-attention or SSM mixer class exists in this file; all three layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) are softmax-attention variants with different KV compression, not linear-complexity recurrent mixers |
| Gated DeltaNet layers | not in its code | not stated | No gated delta-rule recurrent state update is implemented anywhere in the modeling file |
| Mamba-2 layers | not in its code | not stated | No Mamba-2 SSD selective state-space block is implemented in the modeling file |
| Learnable attention sink code only | in its code | not stated | Each attention layer has a per-head learnable sink parameter concatenated as an extra logit column before the softmax, matching gpt-oss… show evidenceEach attention layer has a per-head learnable sink parameter concatenated as an extra logit column before the softmax, matching gpt-oss style sinks |
| Gated attention output | not in its code | not stated | There is no sigmoid gate multiplying the attention output before the output projection; the attention output instead goes through grouped… show the whole verdictThere is no sigmoid gate multiplying the attention output before the output projection; the attention output instead goes through grouped linear projections and a RoPE-conjugate rotation, with no gating mechanism present |
| QK normalization code only | in its code | not stated | Queries are normalized with an unweighted RMSNorm (q_b_norm) per head before the rotary/dot product; keys/KV are normalized with kv_norm… show evidenceQueries are normalized with an unweighted RMSNorm (q_b_norm) per head before the rotary/dot product; keys/KV are normalized with kv_norm as well, both applied before attention |
| Partial RoPE code only | in its code | not stated | RoPE is applied only to the trailing qk_rope_head_dim=64 slice of each head_dim=512 dimension, leaving the leading nope channels… show evidenceRoPE is applied only to the trailing qk_rope_head_dim=64 slice of each head_dim=512 dimension, leaving the leading nope channels untouched, matching MLA's decoupled rope key pattern qk_rope_head_dim = 64head_dim = 512 |
| YaRN RoPE scaling code only | in its code | not stated | rope_scaling type is yarn and is applied to the compress-branch rope used by CSA/HCA attention layers via the ROPE_INIT_FUNCTIONS dispatch… show evidencerope_scaling type is yarn and is applied to the compress-branch rope used by CSA/HCA attention layers via the ROPE_INIT_FUNCTIONS dispatch keyed by rope_type rope_scaling = {"beta_fast": 32, "beta_slow": 1, "factor": 16, "original_max_position_embedd… |
| Layers without positional encoding (NoPE) | not in its code | not stated | Every attention layer applies rotary position embedding (either main theta=10000 or compress theta=160000 with yarn) to its rope slice… show evidenceEvery attention layer applies rotary position embedding (either main theta=10000 or compress theta=160000 with yarn) to its rope slice; there is no layer type that skips positional encoding entirely |
| Mixture of experts code only | in its code | not stated | Feed-forward layers route tokens to a subset of n_routed_experts=384 expert MLPs via a learned router selecting num_experts_per_tok=6… show evidenceFeed-forward layers route tokens to a subset of n_routed_experts=384 expert MLPs via a learned router selecting num_experts_per_tok=6 experts per token n_routed_experts = 384num_experts_per_tok = 6 |
| Shared expert code only | in its code | not stated | Each MoE block also runs a DeepseekV4MLP shared expert that every token passes through, added to the routed output, with n_shared_experts=1 show evidencen_shared_experts = 1 |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | The router adds a per-expert e_score_correction_bias to scores only for top-k selection (not used to weight outputs, which come from the… show evidenceThe router adds a per-expert e_score_correction_bias to scores only for top-k selection (not used to weight outputs, which come from the raw scores gather), matching DeepSeek-V3's noaux_tc topk_method topk_method = "noaux_tc" |
| Multi-token prediction layers code only | in its code | not stated | config declares num_nextn_predict_layers=1 and the model explicitly ignores mtp.* weights on load, indicating the checkpoint carries extra… show evidenceconfig declares num_nextn_predict_layers=1 and the model explicitly ignores mtp.* weights on load, indicating the checkpoint carries extra MTP modules that this config declares a nonzero count of num_nextn_predict_layers = 1 |
| SwiGLU feed-forward code only | in its code | not stated | Both the shared MLP and routed experts compute act_fn(gate)*up with hidden_act=silu, i.e. SwiGLU show evidencehidden_act = "silu" |
multimodal architecture 1
“It is the smallest model in our new architecture family, with native multimodal visual understanding.”
language modelling objective 1
“FIM Completion(Beta)”
filed at the root 1
“With post-training optimization, the new model achieves major improvements in tool usage and intelligent agent tasks”
decoding strategy 2
“DSpark speculative decoding is enabled with a single flag”
“Chat Prefix Completion(Beta)”
reasoning control 1
“The reasoning_effort parameter now supports three levels — low, high, and max — which control how much deliberation the model spends…”
agentic scaffolding 2
“Function Calling”
“JSON Mode”
infrastructure service 1
“When an error occurs in an upstream provider, we can recover by routing to another healthy provider, if your request filters allow it.”
unfiled 4
“The DeepSeek API has innovatively adopted hard disk caching, reducing prices by another order of magnitude.”
“Hybrid reasoning architecture: A single model supports both thinking mode and non-thinking mode”
“how to encode messages in OpenAI-compatible format into input strings for the model”
“we recommend setting the sampling parameters to temperature = 1.0, with top_p = 0.95 for agentic scenarios and top_p = 1.0 otherwise.”
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| api-docs.deepseek.com/updates | official blog | DeepSeek | read |
| arxiv.org/abs/2606.19348 | technical report | DeepSeek | read |
| huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813 | model card | DeepSeek | read |
| openrouter.ai/deepseek/deepseek-v4-pro-0813 | vendor docs | OpenRouter | read |
| lmsys.org/blog/2026-07-06-dspark-sglang | vendor docs | LMSYS | not fetched |