Open Models Almanac
ModelsDeepSeek-V4-Flash

DeepSeek · usage rank #12

DeepSeek-V4-Flash

What DeepSeek-V4-Flash’s own documents say it is built from — every method with the sentence that describes it.

Parameters
284B / 13B active
Context
1M
Attention
sparse (indexer)Compressed Sparse Attention + Heavily Compressed Attention with indexer, sliding window 128, attention sink
Experts
256 routed + 1 shared, top-6
Input
text
Licence
MIT
Released
2026-04-24

Rank by open-weight tokens on OpenRouter, week of 2026-10-26.

Against the consensus recipe

Its documents state 6 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 12 of the 19 architecture features checked — 5 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_deepseek_v4.py, configuration_deepseek_v4.py) and config.json at revision 60d8d70770, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention not in its codenot stated

Attention uses a single shared KV head (shared-KV MQA, num_key_value_heads=1) that is broadcast to all query heads via repeat_kv, but…

show evidence

Attention uses a single shared KV head (shared-KV MQA, num_key_value_heads=1) that is broadcast to all query heads via repeat_kv, but there is no real grouped-head KV projection distinct from MQA and the mechanism here is MLA-adjacent, not classic GQA with a config-selected group size greater than one; since MQA is a degenerate case it is implemented but the architecture is MLA-based rather than GQA-named, still it qualifies technically as MQA — however the attention is MLA-style latent projection, not separate K/V head projections, so GQA as described (separate per-head K/V projections sharing fewer heads) is absent.

num_key_value_heads = 1
self.num_key_value_groups = config.num_attention_heads  # single KV head, broadcast to all
modeling_deepseek_v4.py · L790
Multi-head latent attention code unclearnot stated

Queries use a low-rank q_lora_rank projection and KV is a single shared head (not per-head up-projected from a shared latent with a…

show evidence

Queries use a low-rank q_lora_rank projection and KV is a single shared head (not per-head up-projected from a shared latent with a decoupled RoPE key in the DeepSeek-V2 sense), so this resembles MLA's query side but the KV side is single-head MQA rather than a low-rank latent up-projected per head.

q_lora_rank = 1024num_key_value_heads = 1
self.q_a_proj = nn.Linear(config.hidden_size, config.q_lora_rank, bias=False)
        self.q_a_norm = DeepseekV4RMSNorm(config.q_lora_rank, eps=config.rms_norm_eps)
        self.q_b_proj = nn.Linear(config.q_lora_rank, self.num_heads * self.head_dim, bias=False)
        self.q_b_norm = DeepseekV4UnweightedRMSNorm(eps=config.rms_norm_eps)
        self.kv_proj = nn.Linear(config.hidden_size, self.head_dim, bias=False)
modeling_deepseek_v4.py · L797–801
Sliding-window attention in its codeadopted

Every attention layer maintains a sliding-window KV cache of size config.sliding_window (128) via DynamicSlidingWindowLayer-derived caches…

show evidence

Every attention layer maintains a sliding-window KV cache of size config.sliding_window (128) via DynamicSlidingWindowLayer-derived caches and the attention call passes sliding_window to the interface.

sliding_window = 128
self.keys = full[:, :, -self.sliding_window + 1 :, :]
        self.values = self.keys
modeling_deepseek_v4.py · L199–200
Interleaved sliding-window and global attention code only in its codenot stated

layer_types interleaves sliding_attention (plain local window) layers with compressed_sparse_attention and heavily_compressed_attention…

show evidence

layer_types interleaves sliding_attention (plain local window) layers with compressed_sparse_attention and heavily_compressed_attention layers that attend over long-range compressed history in addition to the local window, giving a fixed interleaving of local vs. extended-context attention per the compress_ratios config.

compress_ratios = [0, 0, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128…
self.rope_layer_type = "main" if self.layer_type == "sliding_attention" else "compress"
modeling_deepseek_v4.py · L788
Indexer-selected sparse attention (DeepSeek Sparse Attention) in its codeadopted

DeepseekV4Indexer scores compressed KV entries per query with a lightweight ReLU-based scorer and the CSA compressor keeps only the top…

show evidence

DeepseekV4Indexer scores compressed KV entries per query with a lightweight ReLU-based scorer and the CSA compressor keeps only the top index_topk entries via topk, matching DeepSeek Sparse Attention.

index_topk = 512index_n_heads = 64index_head_dim = 128
top_k_indices = index_scores.topk(top_k, dim=-1).indices  # [B, S, k]
            invalid = top_k_indices >= causal_threshold.unsqueeze(-1)
            return torch.where(invalid, torch.full_like(top_k_indices, -1), top_k_indices)
modeling_deepseek_v4.py · L593–595
Linear-attention or state-space layers alongside full attention not in its codenot stated

All layer types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) implement softmax attention variants with…

show the whole verdict

All layer types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) implement softmax attention variants with compression/indexing, not linear-complexity recurrent mixers like linear attention or SSMs; no Mamba/linear-attention module is present.

Gated DeltaNet layers not in its codenot stated

No gated delta-rule recurrent state update is implemented anywhere in the modeling file; all token-mixing layers are softmax attention…

show the whole verdict

No gated delta-rule recurrent state update is implemented anywhere in the modeling file; all token-mixing layers are softmax attention with compressors/indexer.

Mamba-2 layers not in its codenot stated

There are no Mamba-2 SSD-style selective state-space blocks in the code; all layers are attention-based with compressors.

Learnable attention sink code only in its codenot stated

Each attention layer has a per-head learnable sink parameter concatenated into the softmax denominator before being dropped, exactly as in…

show evidence

Each attention layer has a per-head learnable sink parameter concatenated into the softmax denominator before being dropped, exactly as in gpt-oss.

sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_deepseek_v4.py · L744–745
self.sinks = nn.Parameter(torch.empty(self.num_heads))
modeling_deepseek_v4.py · L807
Gated attention output not in its codenot stated

The attention output is rotated back (conjugate RoPE) and passed through the grouped output projection, but there is no input-dependent…

show the whole verdict

The attention output is rotated back (conjugate RoPE) and passed through the grouped output projection, but there is no input-dependent sigmoid gate multiplying the attention output.

QK normalization code only in its codenot stated

Queries are normalized with an unweighted RMSNorm (q_b_norm) and the shared KV projection is normalized with kv_norm (DeepseekV4RMSNorm)…

show evidence

Queries are normalized with an unweighted RMSNorm (q_b_norm) and the shared KV projection is normalized with kv_norm (DeepseekV4RMSNorm) before the rotary embedding and dot product.

q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)
modeling_deepseek_v4.py · L828–830
kv = self.kv_norm(self.kv_proj(hidden_states)).view(*hidden_shape).transpose(1, 2)
modeling_deepseek_v4.py · L832
Partial RoPE code only in its codenot stated

RoPE is applied only to the trailing qk_rope_head_dim=64 slice of each 512-dim head, leaving the rest ("nope") channels without position…

show evidence

RoPE is applied only to the trailing qk_rope_head_dim=64 slice of each 512-dim head, leaving the rest ("nope") channels without position signal, controlled by partial_rotary_factor derived from qk_rope_head_dim/head_dim.

qk_rope_head_dim = 64head_dim = 512
rope_dim = cos.shape[-1]
    nope, rope = x[..., :-rope_dim], x[..., -rope_dim:]
    rotated = ((rope.float() * cos) + (rotate_half(rope).float() * sin)).to(x.dtype)
    return torch.cat([nope, rotated], dim=-1)
modeling_deepseek_v4.py · L367–370
YaRN RoPE scaling code only in its codenot stated

rope_scaling type is yarn in the config and compress-branch rope_parameters are built with this yarn configuration (factor 16…

show evidence

rope_scaling type is yarn in the config and compress-branch rope_parameters are built with this yarn configuration (factor 16, beta_fast/slow) consumed by ROPE_INIT_FUNCTIONS for the 'compress' layer type.

rope_scaling = {"beta_fast": 32, "beta_slow": 1, "factor": 16, "original_max_position_embedd…
if compress["rope_type"] == "yarn":
                compress.setdefault("attention_factor", 1.0)
            self.rope_parameters = {"main": main, "compress": compress}
configuration_deepseek_v4.py · L318–320
Layers without positional encoding (NoPE) not in its codenot stated

All attention layers apply partial RoPE to the same trailing slice of each head (the leading slice is always unrotated, not a whole layer…

show evidence

All attention layers apply partial RoPE to the same trailing slice of each head (the leading slice is always unrotated, not a whole layer being without any positional encoding), so there is no layer-level NoPE/RoPE split across whole softmax-attention layers.

qk_rope_head_dim = 64
Mixture of experts in its codeadopted

The SparseMoeBlock routes tokens to a learned top-k subset of n_routed_experts=256 routed experts (num_experts_per_tok=6) via…

show evidence

The SparseMoeBlock routes tokens to a learned top-k subset of n_routed_experts=256 routed experts (num_experts_per_tok=6) via DeepseekV4TopKRouter or a hash-based router.

n_routed_experts = 256num_experts_per_tok = 6
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1094–1095
Shared expert in its codeadopted

Every MoE layer adds the output of a dense DeepseekV4MLP shared expert to the routed-expert output, and n_shared_experts=1 in config…

show evidence

Every MoE layer adds the output of a dense DeepseekV4MLP shared expert to the routed-expert output, and n_shared_experts=1 in config confirms it's enabled.

n_shared_experts = 1
routed = self.experts(flat, indices, weights).view(batch, seq_len, hidden_dim)
        return routed + self.shared_experts(residual)
modeling_deepseek_v4.py · L1147–1148
Auxiliary-loss-free load balancing (selection bias) in its codeadopted

The TopKRouter adds a learned e_score_correction_bias to the scores purely for top-k selection (not used to weight outputs, which are…

show evidence

The TopKRouter adds a learned e_score_correction_bias to the scores purely for top-k selection (not used to weight outputs, which are taken from the un-biased scores), matching DeepSeek-V3's noaux_tc scheme indicated by topk_method.

topk_method = "noaux_tc"
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
        weights = weights / (weights.sum(dim=-1, keepdim=True) + 1e-20)
modeling_deepseek_v4.py · L1094–1096
Multi-token prediction layers in its codeadopted

The config declares num_nextn_predict_layers=1 and the modeling code explicitly ignores unexpected mtp.* weight keys on load, indicating…

show evidence

The config declares num_nextn_predict_layers=1 and the modeling code explicitly ignores unexpected mtp.* weight keys on load, indicating the checkpoint carries MTP modules that this config enables even though they are skipped at load time.

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_deepseek_v4.py · L1266
SwiGLU feed-forward in its codeadopted

Both the dense MLP and the routed Experts module compute act_fn(gate) * up with hidden_act set to silu, i.e. SwiGLU.

show evidencehidden_act = "silu"
gate = self.gate_proj(x).clamp(max=self.limit)
        up = self.up_proj(x).clamp(min=-self.limit, max=self.limit)
        return self.down_proj(self.act_fn(gate) * up)
modeling_deepseek_v4.py · L1033–1035

token mixer 9

softmax attention 3

sliding window attention 1

Sliding Window Attention used

“we introduce a supplementary attention branch to both CSA and HCA in a sliding window manner”

grouped-query attention 1

Multi-query attention used

“CSA then performs core attention in a Multi-Query Attention (MQA) manner”

attention sink 1

Attention sink used

“we employ the trick of attention sink”

sparse attention 5

DeepSeek Sparse Attention core

“Novel Attention: Token-wise compression + DSA (DeepSeek Sparse Attention)”

Compressed Sparse Attention core

“we design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)”

Heavily Compressed Attention core

“we design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)”

Token-wise compression core

“Novel Attention: Token-wise compression + DSA (DeepSeek Sparse Attention)”

Two-stage sparse-attention training used

“the strategy of introducing sparse attention is the same as DeepSeek-V4-Flash, following a two-stage training method.”

hybrid layer stacking 1

Hybrid compressed attention (CSA and HCA) used

“introducing several key architectural innovations: (1) Hybrid Attention Architecture: combines Compressed Sparse Attention (CSA) and…”

channel mixer 10

dense feed-forward network 2

SwiGLU used

“each token-expert pair requires FLOPs (SwiGLU gate, up, and down projections)”

SwiGLU clamping used

“applying SwiGLU clamping effectively eliminates outliers and substantially aids in stabilizing the training process”

mixture of experts 8

Mixture of experts core

“DeepSeek V4 is a Mixture-of-Experts (MoE) language model series”

DeepSeekMoE core

“DeepSeek-V4 series retain the DeepSeekMoE framework”

MegaMoE used

“We have open-sourced the CUDA-based mega-kernel implementation named MegaMoE”

expert routing 3

Hash routing used

“we replace the dense FFN layers in the initial several Transformer blocks with MoE layers that employ Hash routing”

Anticipatory Routing optional

“we "anticipatorily" compute and cache the routing indices to be used later at step”

Loss-spike-triggered Anticipatory Routing optional

“we introduced an automatic detection mechanism that triggers a short rollback and activates Anticipatory Routing exclusively when a loss…”

expert load balancing 1

Auxiliary-loss-free load balancing used

“For auxiliary-loss-free load balancing, we set the bias update speed to 0.001.”

shared experts 1

Shared expert core

“sets fine-grained routed experts and shared experts”

positional encoding 1

Rotary Position Embedding used

“we partially employ the Rotary Positional Embedding (RoPE) to the attention queries, KV entries, and the core attention outputs”

normalization & residual 2

Manifold-Constrained Hyper-Connections core

“DeepSeek-V4 series incorporate Manifold-Constrained Hyper-Connections (mHC) to strengthen the conventional residual connections”

RMSNorm used

“we perform an additional RMSNorm operation on each head of the queries and the only head of the compressed KV entries”

prediction head 1

Multi-Token Prediction core

“DeepSeek-V4 series also set MTP modules and objectives”

context capacity 1

1M-token context window default

“1M context is now the default across all official DeepSeek services”

Not used 1

QK-Clip not used

normalization & residual “we do not employ the QK-Clip technique”

distillation objective 1

Full-vocabulary logit distillation used

“we adopt full-vocabulary logit distillation in our OPD.”

auxiliary loss 1

Sample-level attention masking used

“we employ sample-level attention masking during pre-training.”

optimizer 4

Muon core

“We employ the Muon optimizer for the majority of modules in DeepSeek-V4 series”

AdamW used

“We maintain the AdamW optimizer for the embedding module, the prediction head module, the static biases and gating factors of mHC modules…”

Nesterov momentum used

“we also apply weight decay to Muon parameters, use the Nesterov trick”

Hybrid Newton–Schulz iterations used

“we use hybrid Newton-Schulz iterations for orthogonalization”

learning-rate schedule 1

Batch-size warmup used

“we also employ a batch size scheduling strategy that increases the batch size (in tokens) from a small size to 75.5M”

training precision 1

FP4 and FP8 mixed precision used

“MoE expert parameters use FP4 precision; most other parameters use FP8.”

quantization-aware training 4

Quantization-Aware Training used

“we introduce Quantization-Aware Training (QAT) [QAT] during the post-training stage”

MXFP4 Quantization-Aware Training used

“we incorporate FP4 quantization-aware training for MoE expert weights and the indexer QK path”

FP4 Quantization used

“We apply FP4 (MXFP4) quantization [OCP_MXFormat] to two components”

Straight-Through Estimator used

“equivalent to applying the Straight-Through Estimator (STE) through the quantization operation”

training parallelism 4

All-to-all gradient exchange with local FP32 summation used

“First, an all-to-all operation exchanges local gradients across ranks, and then each rank performs a local sum in FP32.”

Hybrid ZeRO bucket assignment for Muon used

“To address this conflict, we design a hybrid strategy of ZeRO bucket assignment for Muon.”

Knapsack-based balanced assignment of dense parameter matrices used

“we employ a knapsack algorithm to assign parameter matrices to these ranks, ensuring each rank manages a roughly balanced load.”

Modified DualPipe 1F1B pipeline overlap for mHC used

“we adjust the DualPipe 1F1B overlapping scheme to accommodate the increased pipeline communication”

training runtime 3

Tensor-level automatic-differentiation activation checkpointing core

“we implement a tensor-level activation checkpointing mechanism with automatic differentiation support.”

Minimal recomputation-graph extraction by backward graph traversal used

“For each annotated tensor, it performs a backward traversal to identify the minimal subgraph required for its recomputation.”

Selective tensor recomputation for activation-memory reduction used

“we introduce a recomputation strategy that selectively checkpoints intermediate tensors.”

data filtering 1

Filtering batched auto-generated and templated web content used

“we implement filtering strategies to remove batched auto-generated and templated content”

sequence packing 1

Cross-source document packing used

“we pack documents from different sources into appropriate sequences to minimize sample truncation.”

tokenization 1

Quick Instruction tokens used

“We append a set of dedicated special tokens directly to the input sequence, where each token corresponds to a specific auxiliary task.”

reinforcement learning algorithm 1

Group Relative Policy Optimization used

“For the RL stage, we implemented the Group Relative Policy Optimization (GRPO) algorithm”

reward modelling 2

Generative Reward Model used

“we curate rubric-guided RL data and employ a Generative Reward Model (GRM) to evaluate policy trajectories.”

Direct RL optimization of a generative reward model used

“Crucially, we apply RL optimization directly to the GRM itself.”

policy distillation 2

Multi-Teacher On-Policy Distillation core

“we employ multi-teacher On-Policy Distillation (OPD) as the primary technique for merging expert capabilities into the final model.”

On-Policy Distillation used

“we employ multi-teacher On-Policy Distillation (OPD; lu2025onpolicydistillation, minillm) as the primary technique for merging expert…”

reasoning control 3

Persistent reasoning history used

“all reasoning content is fully preserved throughout the entire conversation.”

Think-tag response formatting used

“we utilize specialized response formats demarcated by the <think> and </think> tokens.”

Thinking mode optional

“Both models support 1M context & dual modes (Thinking / Non-Thinking)”

KV cache management 6

Customized heterogeneous KV-cache layout core

“we design a customized KV cache layout.”

Dynamic allocation of fixed-size state-cache pools used

“we pre-allocate a fixed- and limited-size pool of state caches, and dynamically assign it to each sequence.”

KV-cache and sparse-attention-kernel co-design used

“Achieving this requires co-designing the KV cache layout and the sparse attention kernel.”

On-disk KV-cache storage for shared-prefix reuse used

“we leverage an on-disk KV cache storage mechanism to eliminate repeated prefilling for shared-prefix requests.”

Periodic cache checkpointing optional

“This strategy checkpoints SWA KV entries of the last tokens within every tokens, where is a tunable parameter.”

Zero SWA caching optional

“This strategy does not store any SWA KV entries.”

inference quantization 2

FP4 attention-indexer precision used

“attention computation within the lightning indexer is performed in FP4 precision”

FP8 precision used

“FP8 precision is applied to the remaining dimensions”

inference kernel 9

Batch-invariant deterministic kernels core

“we implement end-to-end, bitwise batch-invariant, and deterministic kernels with minimal performance overhead”

DeepGEMM-based batch-invariant matrix multiplication used

“Therefore, we replace it end-to-end with DeepGEMM.”

Distributed shared memory for cross-SM attention exchange used

“the second kernel utilizes distributed shared memory within thread-block clusters, enabling high-speed data exchange across SMs”

Dual-kernel batch-invariant attention decoding used

“To address this, we develop a dual-kernel strategy for batch-invariant decoding.”

Fused mHC kernels used

“we carefully design and implement fused kernels of mHC for both training and inference.”

Host Codegen used

“We mitigate this overhead with Host Codegen, which moves most host-side logic into generated host code”

Per-SM accumulation buffers with deterministic global summation used

“we allocate separate accumulation buffers for each SM, followed by a global deterministic summation across all buffers.”

Separate split-K outputs with deterministic reduction used

“we output each split part separately and perform a deterministic reduction in a subsequent kernel”

Token-order preprocessing and cross-rank buffer isolation used

“we design a token order pre-processing mechanism within each single rank, combined with buffer isolation across multiple ranks.”

agentic scaffolding 3

XML-based tool-call schema with DSML token used

“we introduce a new tool-call schema that employs a special "|DSML|" token and utilizes an XML-based format for tool invocations”

Agentic search used

“the "thinking" mode utilizes agentic search.”

Retrieval-Augmented Search used

“the "non-think" mode employs Retrieval-Augmented Search (RAG)”

Not used 1

Avoiding split-K for batch invariance not used

inference kernel “Therefore, we abandon split-k in most scenarios, which, however, may cause performance degradation.”

infrastructure service 2

Write-ahead log (WAL) used

“we implement a token-granular Write-Ahead Log (WAL) for each generation request.”

Z3 SMT solver integration used

“we integrate the Z3 SMT solver into TileLang’s algebraic system”

unfiled 4

fine-grained EP scheme used

“we introduce a finer-grained expert partitioning scheme... forms a fine-grained pipeline among experts”

TileLang used

“We adopt TileLang to develop a set of fused kernels to replace the vast majority of them”

Two-stage contextual-parallel communication for compressed attention used

“To address these challenges, we design a two-stage communication approach.”

1 more mentioned or evaluated
self-verification evaluated

“candidate natural-language solutions are first generated and filtered by…”

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

SourceKindPublisherStatus
api-docs.deepseek.com/news/news260424official blogDeepSeekread
arxiv.org/abs/2606.19348technical reportDeepSeekread
fe-static.deepseek.com/chat/transparency/deepseek-V4-model-card-EN.pdftechnical reportDeepSeekread
huggingface.co/deepseek-ai/DeepSeek-V4-Flashmodel cardDeepSeekread
huggingface.co/collections/deepseek-ai/deepseek-v4model cardDeepSeeknot fetched
morphllm.com/deepseek-v4-flashthird party analysisMorphnot fetched
↑