Model techniques map
ModelsDeepSeek-V4-Flash

DeepSeek · usage rank #7 · released 2026-04-24

DeepSeek-V4-Flash

What DeepSeek-V4-Flash’s own documents say it is built from — every method with the sentence that describes it.

OpenRouterHugging Face

Curator’s note284B total / 13B active MoE, 1M context, MIT. The arXiv 2606.19348 technical report covers Flash and Pro jointly.

Rank by open-weight tokens on OpenRouter, week of 2026-09-21.

Against the consensus recipe

Its documents state 7 of the 15 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something the documents do not say, not something the model lacks.

  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 13 of the 19 architecture features checked — 6 of them are not stated in its documents.

Documents say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_deepseek_v4.py, configuration_deepseek_v4.py) and config.json at revision 60d8d70770, by anthropic/claude-sonnet-5 on 2026-09-25.

FeatureCodeIts documentsEvidence
Grouped-query attention not in its codenot stated

num_key_value_heads is 1 (a single shared KV head used as MQA within the core attention), but this is implemented as part of MLA-style latent projection with the K=V trick, not classic GQA with an explicit KV projection sized by num_key_value_heads used as a grouping mechanism; regardless, num_key_value_heads=1 means it's MQA-style sharing which is covered by MLA instead

num_key_value_heads = 1
self.num_key_value_groups = config.num_attention_heads  # single KV head, broadcast to all
modeling_deepseek_v4.py · L782
Multi-head latent attention code only in its codenot stated

Queries and KV are both down-projected through low-rank LoRA projections (q_lora_rank, and a shared kv_proj to head_dim) then up-projected, with a decoupled/partial RoPE slice, matching DeepSeek-style MLA.

q_lora_rank = 1024qk_rope_head_dim = 64
self.q_a_proj = nn.Linear(config.hidden_size, config.q_lora_rank, bias=False)
        self.q_a_norm = DeepseekV4RMSNorm(config.q_lora_rank, eps=config.rms_norm_eps)
        self.q_b_proj = nn.Linear(config.q_lora_rank, self.num_heads * self.head_dim, bias=False)
        self.q_b_norm = DeepseekV4UnweightedRMSNorm(eps=config.rms_norm_eps)
        self.kv_proj = nn.Linear(config.hidden_size, self.head_dim, bias=False)
        self.kv_norm = DeepseekV4RMSNorm(self.head_dim, eps=config.rms_norm_eps)
modeling_deepseek_v4.py · L789–794
Sliding-window attention in its code used

sliding_window is set to 128 and is used unconditionally in every attention layer's cache/window mechanism.

sliding_window = 128
self.sliding_window = config.sliding_window
modeling_deepseek_v4.py · L784
Interleaved sliding-window and global attention code only in its codenot stated

layer_types assigns each layer to sliding_attention, compressed_sparse_attention, or heavily_compressed_attention, but all three still apply the same sliding_window cache underneath while only sliding_attention layers lack the additional compressed long-range branch, giving a fixed interleaving of local-only vs local+global-context layers.

sliding_window = 128
self.layer_type = config.layer_types[layer_idx]
        # Sliding-only layers use the "main" (plain θ=10000) rope; CSA/HCA layers
        # share the same yarn-scaled "compress" rope as their compressor.
modeling_deepseek_v4.py · L777–779
Indexer-selected sparse attention (DeepSeek Sparse Attention) in its code core

A DeepseekV4Indexer scores compressed keys with ReLU(q·k) weighted sums and selects the top index_topk=512 entries per query for the CSA compressed-sparse-attention layers.

index_topk = 512index_n_heads = 64index_head_dim = 128
self.index_topk = config.index_topk
        self.kv_proj = nn.Linear(config.hidden_size, 2 * self.head_dim, bias=False)
modeling_deepseek_v4.py · L501–502
top_k_indices = index_scores.topk(top_k, dim=-1).indices  # [B, S, k]
modeling_deepseek_v4.py · L585
Linear-attention or state-space layers alongside full attention not in its codenot stated

All layer types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) use the same softmax DeepseekV4Attention module with a sliding-window cache and optional compressor; there is no linear-attention or SSM token mixer.

Gated DeltaNet layers not in its codenot stated

No gated delta-rule recurrent module exists in the code; all layers use DeepseekV4Attention (softmax attention with optional compression).

Mamba-2 layers not in its codenot stated

No Mamba-2/SSD selective state-space block is implemented anywhere in this file.

Learnable attention sink code only in its codenot stated

Each attention head has a learned sinks parameter concatenated as an extra logit column before the softmax denominator, unconditionally in the eager attention path.

sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_deepseek_v4.py · L736–737
self.sinks = nn.Parameter(torch.empty(self.num_heads))
modeling_deepseek_v4.py · L799
Gated attention output not in its codenot stated

There is no sigmoid gate multiplying the attention output before the output projection; attn_output goes through RoPE-undo and grouped projection only, with no input-dependent gating.

QK normalization code only in its codenot stated

Queries are normalized with an unweighted RMSNorm (q_b_norm) after the up-projection and before RoPE/attention; keys share the kv_norm applied to the shared kv_proj output.

self.q_b_norm = DeepseekV4UnweightedRMSNorm(eps=config.rms_norm_eps)
modeling_deepseek_v4.py · L792
q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)
modeling_deepseek_v4.py · L820–822
Partial RoPE code only in its codenot stated

RoPE is applied only to the trailing qk_rope_head_dim=64 slice of each 512-dim head, with the leading nope portion left untouched, per partial_rotary_factor.

qk_rope_head_dim = 64head_dim = 512
rope_dim = cos.shape[-1]
    nope, rope = x[..., :-rope_dim], x[..., -rope_dim:]
    rotated = ((rope.float() * cos) + (rotate_half(rope).float() * sin)).to(x.dtype)
    return torch.cat([nope, rotated], dim=-1)
modeling_deepseek_v4.py · L359–362
YaRN RoPE scaling code only in its codenot stated

rope_scaling type is yarn with factor 16, applied to the compress-branch rope parameters used by CSA/HCA layers via ROPE_INIT_FUNCTIONS lookup.

rope_scaling = {"beta_fast": 32, "beta_slow": 1, "factor": 16, "original_max_position_embedd…
rope_init_fn = self.compute_default_rope_parameters
            if self.rope_type[layer_type] != "default":
                rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type[layer_type]]
modeling_deepseek_v4.py · L105–107
Layers without positional encoding (NoPE) not in its codenot stated

Every attention layer applies RoPE to its rope slice (partial RoPE on the trailing qk_rope_head_dim channels) regardless of layer_type; there is no layer that skips positional encoding entirely.

cos, sin = position_embeddings[self.rope_layer_type]

        q_residual = self.q_a_norm(self.q_a_proj(hidden_states))
        q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)
modeling_deepseek_v4.py · L817–822
Mixture of experts in its code core

Feed-forward layers route tokens to top num_experts_per_tok=6 of n_routed_experts=256 experts via a learned TopKRouter (or hash router for early layers).

n_routed_experts = 256num_experts_per_tok = 6
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
modeling_deepseek_v4.py · L1086
Shared expert in its code core

Each SparseMoeBlock adds a DeepseekV4MLP shared expert output to every token's routed output, and n_shared_experts is 1 (nonzero).

n_shared_experts = 1
routed = self.experts(flat, indices, weights).view(batch, seq_len, hidden_dim)
        return routed + self.shared_experts(residual)
modeling_deepseek_v4.py · L1139–1140
Auxiliary-loss-free load balancing (selection bias) in its code used

The router adds e_score_correction_bias to scores solely for top-k selection (not for the output weights, which are gathered from the unbiased scores), matching DeepSeek-V3's noaux_tc method.

topk_method = "noaux_tc"
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1086–1087
Multi-token prediction layers in its code core

num_nextn_predict_layers is 1 in the config, and the modeling file explicitly declares an ignore pattern for mtp weights on load, indicating declared but skipped MTP modules.

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_deepseek_v4.py · L1259
SwiGLU feed-forward in its code used

Both the dense MLP and the MoE experts compute act_fn(gate)*up with hidden_act=silu, i.e. SwiGLU.

hidden_act = "silu"
gate = self.gate_proj(x).clamp(max=self.limit)
        up = self.up_proj(x).clamp(min=-self.limit, max=self.limit)
        return self.down_proj(self.act_fn(gate) * up)
modeling_deepseek_v4.py · L1025–1027

model architecture 25

token mixer 9

softmax attention 3

sliding window attention 1

Sliding Window Attention used

grouped-query attention 1

Multi-query attention used

attention sink 1

Attention sink used

hybrid layer stacking 1

Hybrid Attention used

channel mixer 10

dense feed-forward network 2

SwiGLU used
SwiGLU clamping used

mixture of experts 8

Mixture of Experts core
DeepSeekMoE core
MegaMoE used

expert load balancing 1

Auxiliary-loss-free load balancing used

shared experts 1

Shared Experts core

positional encoding 1

Rotary Position Embedding used

normalization & residual 3

Manifold-Constrained Hyper-Connections core
RMSNorm used
QK-Clip not used

prediction head 1

Multi-Token Prediction core

context capacity 1

1M-token context window default

training objective 2

distillation objective 1

Full-vocabulary logit distillation used

auxiliary loss 1

Sample-level attention masking used

optimization 17

optimizer 4

Muon core
AdamW used
Nesterov momentum used
Hybrid Newton–Schulz iterations used

learning-rate schedule 1

Scheduled batch-size growth used

training precision 1

FP4+FP8 mixed precision used

quantization-aware training 4

Quantization-Aware Training used
FP4 Quantization used
FP4 Quantization-Aware Training used
Straight-Through Estimator used

training parallelism 4

All-to-all Gradient Exchange with Local FP32 Summation used
Hybrid ZeRO Bucket Assignment for Muon used
Knapsack-Based Balanced Assignment of Dense Parameter Matrices used
Modified DualPipe 1F1B Pipeline Overlap for mHC used

training runtime 3

Tensor-level automatic-differentiation activation checkpointing core
Minimal recomputation-graph extraction by backward traversal used
Selective tensor recomputation for activation-memory reduction used

data curation 3

data filtering 1

Filtering batched auto-generated and templated content used

sequence packing 1

Cross-source document packing used

tokenization 1

Quick Instruction tokens used

post-training 5

reinforcement learning algorithm 1

Group Relative Policy Optimization used

reward modelling 2

Generative Reward Model used
Direct RL optimization of a generative reward model used

policy distillation 2

Multi-Teacher On-Policy Distillation core
On-Policy Distillation used

inference & serving 24

reasoning control 3

Cross-turn persistent reasoning history used
Think-tag response formatting used
Thinking mode selection optional

KV cache management 6

Customized heterogeneous KV-cache layout core
Dynamic allocation of fixed-size state-cache pools used
KV-cache and sparse-attention-kernel co-design used
On-disk KV-cache storage used
Periodic cache checkpointing optional
Zero SWA caching optional

inference quantization 2

FP8 KV-cache quantization used
FP4 precision for the attention indexer used

inference kernel 10

Batch-invariant deterministic kernels core
DeepGEMM-based batch-invariant matrix multiplication used
Distributed shared memory for cross-SM attention data exchange used
Dual-kernel batch-invariant attention decoding used
Fused mHC kernels used
Host Codegen used
Per-SM accumulation buffers with deterministic global summation used
Separate split-K outputs with deterministic reduction used
Token-order preprocessing and cross-rank buffer isolation for deterministic MoE backward used
Avoiding split-K not used

agentic scaffolding 3

XML-based tool-call schema with DSML token used
Agentic search used
Retrieval-Augmented Search used

software implementation 2

infrastructure service 2

Write-ahead log used
Z3 SMT solver integration used

unfiled 4

fine-grained EP scheme used
TileLang used
Two-stage contextual-parallel communication for compressed attention used
self-verification evaluated

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated document list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

DocumentKindPublisherStatus
api-docs.deepseek.com/news/news260424official blogDeepSeekread
arxiv.org/abs/2606.19348technical reportDeepSeekread
fe-static.deepseek.com/chat/transparency/deepseek-V4-model-card-EN.pdftechnical reportDeepSeekread
huggingface.co/deepseek-ai/DeepSeek-V4-Flashmodel cardDeepSeekread
huggingface.co/collections/deepseek-ai/deepseek-v4model cardDeepSeeknot fetched
morphllm.com/deepseek-v4-flashthird party analysisMorphnot fetched