Open Models Almanac
ModelsDeepSeek-V4-Pro-0813

DeepSeek

DeepSeek-V4-Pro-0813

What DeepSeek-V4-Pro-0813’s own documents say it is built from — every method with the sentence that describes it.

Parameters
1.6T / 49B active
Context
1M
Attention
sparse (indexer)Compressed Sparse Attention + Heavily Compressed Attention with indexer, sliding window 128, attention sink
Experts
384 routed + 1 shared, top-6
Input
text
Licence
MIT
Released
2026-08-13

Against the consensus recipe

Its documents state 1 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 12 of the 19 architecture features checked — 12 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_deepseek_v4.py, configuration_deepseek_v4.py) and config.json at revision 72e1d3230f, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention not in its codenot stated

num_key_value_heads=1 with the single KV head broadcast to all query heads is multi-query attention implemented via repeat_kv, but the…

show evidence

num_key_value_heads=1 with the single KV head broadcast to all query heads is multi-query attention implemented via repeat_kv, but the attention itself is MLA (latent KV projection) rather than classic GQA key/value projections, and the check explicitly excludes MLA-style attention

num_key_value_heads = 1
self.num_key_value_groups = config.num_attention_heads  # single KV head, broadcast to all
modeling_deepseek_v4.py · L790
Multi-head latent attention code only in its codenot stated

Queries and KV are down-projected through low-rank lora ranks (q_lora_rank, kv via kv_proj/kv_norm) and up-projected per head…

show evidence

Queries and KV are down-projected through low-rank lora ranks (q_lora_rank, kv via kv_proj/kv_norm) and up-projected per head, DeepSeek-style, with a decoupled rope key applied via partial rotary

q_lora_rank = 1536qk_rope_head_dim = 64
self.q_a_proj = nn.Linear(config.hidden_size, config.q_lora_rank, bias=False)
        self.q_a_norm = DeepseekV4RMSNorm(config.q_lora_rank, eps=config.rms_norm_eps)
        self.q_b_proj = nn.Linear(config.q_lora_rank, self.num_heads * self.head_dim, bias=False)
        self.q_b_norm = DeepseekV4UnweightedRMSNorm(eps=config.rms_norm_eps)
        self.kv_proj = nn.Linear(config.hidden_size, self.head_dim, bias=False)
        self.kv_norm = DeepseekV4RMSNorm(self.head_dim, eps=config.rms_norm_eps)
modeling_deepseek_v4.py · L797–802
Sliding-window attention code only in its codenot stated

Every attention layer uses a sliding-window KV cache (DynamicSlidingWindowLayer) with sliding_window=128 passed to the attention interface

show evidencesliding_window = 128
self.sliding_window = config.sliding_window
modeling_deepseek_v4.py · L792
sliding_window=self.sliding_window,
modeling_deepseek_v4.py · L868
Interleaved sliding-window and global attention not in its codenot stated

All layers apply the same sliding-window K cache regardless of layer_types (sliding_attention, compressed_sparse_attention…

show evidence

All layers apply the same sliding-window K cache regardless of layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention all share the sliding-window branch); the compressor variants add extra long-range compressed KV on top, but none attend to the literal full context without a window, so there is no interleaving of sliding vs full/global attention

sliding_window = 128
if past_key_values is not None:  # sliding where K==V
            kv = past_key_values.update(kv, kv, self.layer_idx)[0]
modeling_deepseek_v4.py · L835–836
Indexer-selected sparse attention (DeepSeek Sparse Attention) code only in its codenot stated

DeepseekV4Indexer scores compressed KV entries with ReLU(q·k) weighted sums and the CSA compressor selects the top index_topk=1024 entries…

show evidence

DeepseekV4Indexer scores compressed KV entries with ReLU(q·k) weighted sums and the CSA compressor selects the top index_topk=1024 entries per query via index_scores.topk

index_topk = 1024index_n_heads = 64index_head_dim = 128
scores = torch.matmul(q.float(), compressed_kv.transpose(-1, -2).float().unsqueeze(1))  # [B, S, H, T]
        scores = F.relu(scores) * self.softmax_scale
        weights = self.weights_proj(hidden_states).float() * self.weights_scaling  # [B, S, H]
        return (scores * weights.unsqueeze(-1)).sum(dim=2)  # [B, S, T]
modeling_deepseek_v4.py · L467–470
top_k_indices = index_scores.topk(top_k, dim=-1).indices  # [B, S, k]
modeling_deepseek_v4.py · L593
Linear-attention or state-space layers alongside full attention not in its codenot stated

No recurrent linear-attention or SSM mixer class exists in this file; all three layer_types (sliding_attention…

show the whole verdict

No recurrent linear-attention or SSM mixer class exists in this file; all three layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) are softmax-attention variants with different KV compression, not linear-complexity recurrent mixers

Gated DeltaNet layers not in its codenot stated

No gated delta-rule recurrent state update is implemented anywhere in the modeling file

Mamba-2 layers not in its codenot stated

No Mamba-2 SSD selective state-space block is implemented in the modeling file

Learnable attention sink code only in its codenot stated

Each attention layer has a per-head learnable sink parameter concatenated as an extra logit column before the softmax, matching gpt-oss…

show evidence

Each attention layer has a per-head learnable sink parameter concatenated as an extra logit column before the softmax, matching gpt-oss style sinks

self.sinks = nn.Parameter(torch.empty(self.num_heads))
modeling_deepseek_v4.py · L807
sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_deepseek_v4.py · L744–745
Gated attention output not in its codenot stated

There is no sigmoid gate multiplying the attention output before the output projection; the attention output instead goes through grouped…

show the whole verdict

There is no sigmoid gate multiplying the attention output before the output projection; the attention output instead goes through grouped linear projections and a RoPE-conjugate rotation, with no gating mechanism present

QK normalization code only in its codenot stated

Queries are normalized with an unweighted RMSNorm (q_b_norm) per head before the rotary/dot product; keys/KV are normalized with kv_norm…

show evidence

Queries are normalized with an unweighted RMSNorm (q_b_norm) per head before the rotary/dot product; keys/KV are normalized with kv_norm as well, both applied before attention

q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)

        kv = self.kv_norm(self.kv_proj(hidden_states)).view(*hidden_shape).transpose(1, 2)
modeling_deepseek_v4.py · L828–832
Partial RoPE code only in its codenot stated

RoPE is applied only to the trailing qk_rope_head_dim=64 slice of each head_dim=512 dimension, leaving the leading nope channels…

show evidence

RoPE is applied only to the trailing qk_rope_head_dim=64 slice of each head_dim=512 dimension, leaving the leading nope channels untouched, matching MLA's decoupled rope key pattern

qk_rope_head_dim = 64head_dim = 512
rope_dim = cos.shape[-1]
    nope, rope = x[..., :-rope_dim], x[..., -rope_dim:]
    rotated = ((rope.float() * cos) + (rotate_half(rope).float() * sin)).to(x.dtype)
    return torch.cat([nope, rotated], dim=-1)
modeling_deepseek_v4.py · L367–370
YaRN RoPE scaling code only in its codenot stated

rope_scaling type is yarn and is applied to the compress-branch rope used by CSA/HCA attention layers via the ROPE_INIT_FUNCTIONS dispatch…

show evidence

rope_scaling type is yarn and is applied to the compress-branch rope used by CSA/HCA attention layers via the ROPE_INIT_FUNCTIONS dispatch keyed by rope_type

rope_scaling = {"beta_fast": 32, "beta_slow": 1, "factor": 16, "original_max_position_embedd…
compress.setdefault("rope_type", "default")
            if compress["rope_type"] == "yarn":
                compress.setdefault("attention_factor", 1.0)
configuration_deepseek_v4.py · L317–319
rope_init_fn = self.compute_default_rope_parameters
            if self.rope_type[layer_type] != "default":
                rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type[layer_type]]
modeling_deepseek_v4.py · L104–106
Layers without positional encoding (NoPE) not in its codenot stated

Every attention layer applies rotary position embedding (either main theta=10000 or compress theta=160000 with yarn) to its rope slice…

show evidence

Every attention layer applies rotary position embedding (either main theta=10000 or compress theta=160000 with yarn) to its rope slice; there is no layer type that skips positional encoding entirely

self.rope_layer_type = "main" if self.layer_type == "sliding_attention" else "compress"
modeling_deepseek_v4.py · L788
Mixture of experts code only in its codenot stated

Feed-forward layers route tokens to a subset of n_routed_experts=384 expert MLPs via a learned router selecting num_experts_per_tok=6…

show evidence

Feed-forward layers route tokens to a subset of n_routed_experts=384 expert MLPs via a learned router selecting num_experts_per_tok=6 experts per token

n_routed_experts = 384num_experts_per_tok = 6
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1094–1095
Shared expert code only in its codenot stated

Each MoE block also runs a DeepseekV4MLP shared expert that every token passes through, added to the routed output, with n_shared_experts=1

show evidencen_shared_experts = 1
self.shared_experts = DeepseekV4MLP(config)
modeling_deepseek_v4.py · L1137
routed = self.experts(flat, indices, weights).view(batch, seq_len, hidden_dim)
        return routed + self.shared_experts(residual)
modeling_deepseek_v4.py · L1147–1148
Auxiliary-loss-free load balancing (selection bias) code only in its codenot stated

The router adds a per-expert e_score_correction_bias to scores only for top-k selection (not used to weight outputs, which come from the…

show evidence

The router adds a per-expert e_score_correction_bias to scores only for top-k selection (not used to weight outputs, which come from the raw scores gather), matching DeepSeek-V3's noaux_tc topk_method

topk_method = "noaux_tc"
self.e_score_correction_bias = nn.Buffer(torch.zeros(self.num_experts))
modeling_deepseek_v4.py · L1088
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1094–1095
Multi-token prediction layers code only in its codenot stated

config declares num_nextn_predict_layers=1 and the model explicitly ignores mtp.* weights on load, indicating the checkpoint carries extra…

show evidence

config declares num_nextn_predict_layers=1 and the model explicitly ignores mtp.* weights on load, indicating the checkpoint carries extra MTP modules that this config declares a nonzero count of

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_deepseek_v4.py · L1266
SwiGLU feed-forward code only in its codenot stated

Both the shared MLP and routed experts compute act_fn(gate)*up with hidden_act=silu, i.e. SwiGLU

show evidencehidden_act = "silu"
gate = self.gate_proj(x).clamp(max=self.limit)
        up = self.up_proj(x).clamp(min=-self.limit, max=self.limit)
        return self.down_proj(self.act_fn(gate) * up)
modeling_deepseek_v4.py · L1033–1035

multimodal architecture 1

Native multimodal architecture core

“It is the smallest model in our new architecture family, with native multimodal visual understanding.”

language modelling objective 1

Fill-in-the-middle completion optional

“FIM Completion(Beta)”

filed at the root 1

Post-training optimization used

“With post-training optimization, the new model achieves major improvements in tool usage and intelligent agent tasks”

decoding strategy 2

DSpark optional

“DSpark speculative decoding is enabled with a single flag”

Chat Prefix Completion optional

“Chat Prefix Completion(Beta)”

reasoning control 1

Thinking mode optional

“The reasoning_effort parameter now supports three levels — low, high, and max — which control how much deliberation the model spends…”

agentic scaffolding 2

Function calling optional

“Function Calling”

JSON mode optional

“JSON Mode”

infrastructure service 1

Automatic provider failover and retry optional

“When an error occurs in an upstream provider, we can recover by routing to another healthy provider, if your request filters allow it.”

unfiled 4

disk-based context caching core

“The DeepSeek API has innovatively adopted hard disk caching, reducing prices by another order of magnitude.”

hybrid reasoning architecture core

“Hybrid reasoning architecture: A single model supports both thinking mode and non-thinking mode”

OpenAI-compatible message encoding used

“how to encode messages in OpenAI-compatible format into input strings for the model”

recommended sampling parameters used

“we recommend setting the sampling parameters to temperature = 1.0, with top_p = 0.95 for agentic scenarios and top_p = 1.0 otherwise.”

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

SourceKindPublisherStatus
api-docs.deepseek.com/updatesofficial blogDeepSeekread
arxiv.org/abs/2606.19348technical reportDeepSeekread
huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813model cardDeepSeekread
openrouter.ai/deepseek/deepseek-v4-pro-0813vendor docsOpenRouterread
lmsys.org/blog/2026-07-06-dspark-sglangvendor docsLMSYSnot fetched
↑