DeepSeek · usage rank #12
DeepSeek-V4-Flash
What DeepSeek-V4-Flash’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 284B / 13B active
- Context
- 1M
- Attention
- sparse (indexer)Compressed Sparse Attention + Heavily Compressed Attention with indexer, sliding window 128, attention sink
- Experts
- 256 routed + 1 shared, top-6
- Input
- text
- Licence
- MIT
- Released
- 2026-04-24
Rank by open-weight tokens on OpenRouter, week of 2026-10-26.
Against the consensus recipe
Its documents state 6 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictioncore
- Grouped-query attentionnot in its documents
Optimization
- Muoncore
- Expert parallelismnot in its documents
- Quantization-Aware Trainingused
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationcore
Inference & serving
- Thinking modeoptional
- Multi-Token Predictionnot in its documents
- FP8not in its documents
Checked in its code
Its code shows 12 of the 19 architecture features checked — 5 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_deepseek_v4.py, configuration_deepseek_v4.py) and config.json at revision 60d8d70770, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention | not in its code | not stated | Attention uses a single shared KV head (shared-KV MQA, num_key_value_heads=1) that is broadcast to all query heads via repeat_kv, but… show evidenceAttention uses a single shared KV head (shared-KV MQA, num_key_value_heads=1) that is broadcast to all query heads via repeat_kv, but there is no real grouped-head KV projection distinct from MQA and the mechanism here is MLA-adjacent, not classic GQA with a config-selected group size greater than one; since MQA is a degenerate case it is implemented but the architecture is MLA-based rather than GQA-named, still it qualifies technically as MQA — however the attention is MLA-style latent projection, not separate K/V head projections, so GQA as described (separate per-head K/V projections sharing fewer heads) is absent. num_key_value_heads = 1 |
| Multi-head latent attention | code unclear | not stated | Queries use a low-rank q_lora_rank projection and KV is a single shared head (not per-head up-projected from a shared latent with a… show evidenceQueries use a low-rank q_lora_rank projection and KV is a single shared head (not per-head up-projected from a shared latent with a decoupled RoPE key in the DeepSeek-V2 sense), so this resembles MLA's query side but the KV side is single-head MQA rather than a low-rank latent up-projected per head. q_lora_rank = 1024num_key_value_heads = 1 |
| Sliding-window attention | in its code | adopted | Every attention layer maintains a sliding-window KV cache of size config.sliding_window (128) via DynamicSlidingWindowLayer-derived caches… show evidenceEvery attention layer maintains a sliding-window KV cache of size config.sliding_window (128) via DynamicSlidingWindowLayer-derived caches and the attention call passes sliding_window to the interface. sliding_window = 128 |
| Interleaved sliding-window and global attention code only | in its code | not stated | layer_types interleaves sliding_attention (plain local window) layers with compressed_sparse_attention and heavily_compressed_attention… show evidencelayer_types interleaves sliding_attention (plain local window) layers with compressed_sparse_attention and heavily_compressed_attention layers that attend over long-range compressed history in addition to the local window, giving a fixed interleaving of local vs. extended-context attention per the compress_ratios config. compress_ratios = [0, 0, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128… |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | in its code | adopted | DeepseekV4Indexer scores compressed KV entries per query with a lightweight ReLU-based scorer and the CSA compressor keeps only the top… show evidenceDeepseekV4Indexer scores compressed KV entries per query with a lightweight ReLU-based scorer and the CSA compressor keeps only the top index_topk entries via topk, matching DeepSeek Sparse Attention. index_topk = 512index_n_heads = 64index_head_dim = 128 |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | All layer types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) implement softmax attention variants with… show the whole verdictAll layer types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) implement softmax attention variants with compression/indexing, not linear-complexity recurrent mixers like linear attention or SSMs; no Mamba/linear-attention module is present. |
| Gated DeltaNet layers | not in its code | not stated | No gated delta-rule recurrent state update is implemented anywhere in the modeling file; all token-mixing layers are softmax attention… show the whole verdictNo gated delta-rule recurrent state update is implemented anywhere in the modeling file; all token-mixing layers are softmax attention with compressors/indexer. |
| Mamba-2 layers | not in its code | not stated | There are no Mamba-2 SSD-style selective state-space blocks in the code; all layers are attention-based with compressors. |
| Learnable attention sink code only | in its code | not stated | Each attention layer has a per-head learnable sink parameter concatenated into the softmax denominator before being dropped, exactly as in… show evidenceEach attention layer has a per-head learnable sink parameter concatenated into the softmax denominator before being dropped, exactly as in gpt-oss. |
| Gated attention output | not in its code | not stated | The attention output is rotated back (conjugate RoPE) and passed through the grouped output projection, but there is no input-dependent… show the whole verdictThe attention output is rotated back (conjugate RoPE) and passed through the grouped output projection, but there is no input-dependent sigmoid gate multiplying the attention output. |
| QK normalization code only | in its code | not stated | Queries are normalized with an unweighted RMSNorm (q_b_norm) and the shared KV projection is normalized with kv_norm (DeepseekV4RMSNorm)… show evidenceQueries are normalized with an unweighted RMSNorm (q_b_norm) and the shared KV projection is normalized with kv_norm (DeepseekV4RMSNorm) before the rotary embedding and dot product. |
| Partial RoPE code only | in its code | not stated | RoPE is applied only to the trailing qk_rope_head_dim=64 slice of each 512-dim head, leaving the rest ("nope") channels without position… show evidenceRoPE is applied only to the trailing qk_rope_head_dim=64 slice of each 512-dim head, leaving the rest ("nope") channels without position signal, controlled by partial_rotary_factor derived from qk_rope_head_dim/head_dim. qk_rope_head_dim = 64head_dim = 512 |
| YaRN RoPE scaling code only | in its code | not stated | rope_scaling type is yarn in the config and compress-branch rope_parameters are built with this yarn configuration (factor 16… show evidencerope_scaling type is yarn in the config and compress-branch rope_parameters are built with this yarn configuration (factor 16, beta_fast/slow) consumed by ROPE_INIT_FUNCTIONS for the 'compress' layer type. rope_scaling = {"beta_fast": 32, "beta_slow": 1, "factor": 16, "original_max_position_embedd… |
| Layers without positional encoding (NoPE) | not in its code | not stated | All attention layers apply partial RoPE to the same trailing slice of each head (the leading slice is always unrotated, not a whole layer… show evidenceAll attention layers apply partial RoPE to the same trailing slice of each head (the leading slice is always unrotated, not a whole layer being without any positional encoding), so there is no layer-level NoPE/RoPE split across whole softmax-attention layers. qk_rope_head_dim = 64 |
| Mixture of experts | in its code | adopted | The SparseMoeBlock routes tokens to a learned top-k subset of n_routed_experts=256 routed experts (num_experts_per_tok=6) via… show evidenceThe SparseMoeBlock routes tokens to a learned top-k subset of n_routed_experts=256 routed experts (num_experts_per_tok=6) via DeepseekV4TopKRouter or a hash-based router. n_routed_experts = 256num_experts_per_tok = 6 |
| Shared expert | in its code | adopted | Every MoE layer adds the output of a dense DeepseekV4MLP shared expert to the routed-expert output, and n_shared_experts=1 in config… show evidenceEvery MoE layer adds the output of a dense DeepseekV4MLP shared expert to the routed-expert output, and n_shared_experts=1 in config confirms it's enabled. n_shared_experts = 1 |
| Auxiliary-loss-free load balancing (selection bias) | in its code | adopted | The TopKRouter adds a learned e_score_correction_bias to the scores purely for top-k selection (not used to weight outputs, which are… show evidenceThe TopKRouter adds a learned e_score_correction_bias to the scores purely for top-k selection (not used to weight outputs, which are taken from the un-biased scores), matching DeepSeek-V3's noaux_tc scheme indicated by topk_method. topk_method = "noaux_tc" |
| Multi-token prediction layers | in its code | adopted | The config declares num_nextn_predict_layers=1 and the modeling code explicitly ignores unexpected mtp.* weight keys on load, indicating… show evidenceThe config declares num_nextn_predict_layers=1 and the modeling code explicitly ignores unexpected mtp.* weight keys on load, indicating the checkpoint carries MTP modules that this config enables even though they are skipped at load time. num_nextn_predict_layers = 1 |
| SwiGLU feed-forward | in its code | adopted | Both the dense MLP and the routed Experts module compute act_fn(gate) * up with hidden_act set to silu, i.e. SwiGLU. show evidencehidden_act = "silu" |
token mixer 9
softmax attention 3
sliding window attention 1
“we introduce a supplementary attention branch to both CSA and HCA in a sliding window manner”
grouped-query attention 1
“CSA then performs core attention in a Multi-Query Attention (MQA) manner”
attention sink 1
“we employ the trick of attention sink”
sparse attention 5
“Novel Attention: Token-wise compression + DSA (DeepSeek Sparse Attention)”
“we design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)”
“we design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)”
“Novel Attention: Token-wise compression + DSA (DeepSeek Sparse Attention)”
“the strategy of introducing sparse attention is the same as DeepSeek-V4-Flash, following a two-stage training method.”
hybrid layer stacking 1
“introducing several key architectural innovations: (1) Hybrid Attention Architecture: combines Compressed Sparse Attention (CSA) and…”
channel mixer 10
dense feed-forward network 2
“each token-expert pair requires FLOPs (SwiGLU gate, up, and down projections)”
“applying SwiGLU clamping effectively eliminates outliers and substantially aids in stabilizing the training process”
mixture of experts 8
“DeepSeek V4 is a Mixture-of-Experts (MoE) language model series”
“DeepSeek-V4 series retain the DeepSeekMoE framework”
“We have open-sourced the CUDA-based mega-kernel implementation named MegaMoE”
expert routing 3
“we replace the dense FFN layers in the initial several Transformer blocks with MoE layers that employ Hash routing”
“we "anticipatorily" compute and cache the routing indices to be used later at step”
“we introduced an automatic detection mechanism that triggers a short rollback and activates Anticipatory Routing exclusively when a loss…”
expert load balancing 1
“For auxiliary-loss-free load balancing, we set the bias update speed to 0.001.”
positional encoding 1
“we partially employ the Rotary Positional Embedding (RoPE) to the attention queries, KV entries, and the core attention outputs”
normalization & residual 2
“DeepSeek-V4 series incorporate Manifold-Constrained Hyper-Connections (mHC) to strengthen the conventional residual connections”
“we perform an additional RMSNorm operation on each head of the queries and the only head of the compressed KV entries”
prediction head 1
“DeepSeek-V4 series also set MTP modules and objectives”
context capacity 1
“1M context is now the default across all official DeepSeek services”
Not used 1
normalization & residual “we do not employ the QK-Clip technique”
distillation objective 1
“we adopt full-vocabulary logit distillation in our OPD.”
auxiliary loss 1
“we employ sample-level attention masking during pre-training.”
optimization 17
optimizer 4
“We employ the Muon optimizer for the majority of modules in DeepSeek-V4 series”
“We maintain the AdamW optimizer for the embedding module, the prediction head module, the static biases and gating factors of mHC modules…”
“we also apply weight decay to Muon parameters, use the Nesterov trick”
“we use hybrid Newton-Schulz iterations for orthogonalization”
learning-rate schedule 1
“we also employ a batch size scheduling strategy that increases the batch size (in tokens) from a small size to 75.5M”
training precision 1
“MoE expert parameters use FP4 precision; most other parameters use FP8.”
quantization-aware training 4
“we introduce Quantization-Aware Training (QAT) [QAT] during the post-training stage”
“we incorporate FP4 quantization-aware training for MoE expert weights and the indexer QK path”
“We apply FP4 (MXFP4) quantization [OCP_MXFormat] to two components”
“equivalent to applying the Straight-Through Estimator (STE) through the quantization operation”
training parallelism 4
“First, an all-to-all operation exchanges local gradients across ranks, and then each rank performs a local sum in FP32.”
“To address this conflict, we design a hybrid strategy of ZeRO bucket assignment for Muon.”
“we employ a knapsack algorithm to assign parameter matrices to these ranks, ensuring each rank manages a roughly balanced load.”
“we adjust the DualPipe 1F1B overlapping scheme to accommodate the increased pipeline communication”
training runtime 3
“we implement a tensor-level activation checkpointing mechanism with automatic differentiation support.”
“For each annotated tensor, it performs a backward traversal to identify the minimal subgraph required for its recomputation.”
“we introduce a recomputation strategy that selectively checkpoints intermediate tensors.”
data filtering 1
“we implement filtering strategies to remove batched auto-generated and templated content”
sequence packing 1
“we pack documents from different sources into appropriate sequences to minimize sample truncation.”
tokenization 1
“We append a set of dedicated special tokens directly to the input sequence, where each token corresponds to a specific auxiliary task.”
reinforcement learning algorithm 1
“For the RL stage, we implemented the Group Relative Policy Optimization (GRPO) algorithm”
reward modelling 2
“we curate rubric-guided RL data and employ a Generative Reward Model (GRM) to evaluate policy trajectories.”
“Crucially, we apply RL optimization directly to the GRM itself.”
policy distillation 2
“we employ multi-teacher On-Policy Distillation (OPD) as the primary technique for merging expert capabilities into the final model.”
“we employ multi-teacher On-Policy Distillation (OPD; lu2025onpolicydistillation, minillm) as the primary technique for merging expert…”
reasoning control 3
“all reasoning content is fully preserved throughout the entire conversation.”
“we utilize specialized response formats demarcated by the <think> and </think> tokens.”
“Both models support 1M context & dual modes (Thinking / Non-Thinking)”
KV cache management 6
“we design a customized KV cache layout.”
“we pre-allocate a fixed- and limited-size pool of state caches, and dynamically assign it to each sequence.”
“Achieving this requires co-designing the KV cache layout and the sparse attention kernel.”
“we leverage an on-disk KV cache storage mechanism to eliminate repeated prefilling for shared-prefix requests.”
“This strategy checkpoints SWA KV entries of the last tokens within every tokens, where is a tunable parameter.”
“This strategy does not store any SWA KV entries.”
inference quantization 2
“attention computation within the lightning indexer is performed in FP4 precision”
“FP8 precision is applied to the remaining dimensions”
inference kernel 9
“we implement end-to-end, bitwise batch-invariant, and deterministic kernels with minimal performance overhead”
“Therefore, we replace it end-to-end with DeepGEMM.”
“the second kernel utilizes distributed shared memory within thread-block clusters, enabling high-speed data exchange across SMs”
“To address this, we develop a dual-kernel strategy for batch-invariant decoding.”
“we carefully design and implement fused kernels of mHC for both training and inference.”
“We mitigate this overhead with Host Codegen, which moves most host-side logic into generated host code”
“we allocate separate accumulation buffers for each SM, followed by a global deterministic summation across all buffers.”
“we output each split part separately and perform a deterministic reduction in a subsequent kernel”
“we design a token order pre-processing mechanism within each single rank, combined with buffer isolation across multiple ranks.”
agentic scaffolding 3
“we introduce a new tool-call schema that employs a special "|DSML|" token and utilizes an XML-based format for tool invocations”
“the "thinking" mode utilizes agentic search.”
“the "non-think" mode employs Retrieval-Augmented Search (RAG)”
Not used 1
inference kernel “Therefore, we abandon split-k in most scenarios, which, however, may cause performance degradation.”
infrastructure service 2
“we implement a token-granular Write-Ahead Log (WAL) for each generation request.”
“we integrate the Z3 SMT solver into TileLang’s algebraic system”
unfiled 4
“we introduce a finer-grained expert partitioning scheme... forms a fine-grained pipeline among experts”
“We adopt TileLang to develop a set of fused kernels to replace the vast majority of them”
“To address these challenges, we design a two-stage communication approach.”
1 more mentioned or evaluated
“candidate natural-language solutions are first generated and filtered by…”
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- DeepSeek-V4-Flash architecture card — LLM Architecture Gallery explainer sebastianraschka.com
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| api-docs.deepseek.com/news/news260424 | official blog | DeepSeek | read |
| arxiv.org/abs/2606.19348 | technical report | DeepSeek | read |
| fe-static.deepseek.com/chat/transparency/deepseek-V4-model-card-EN.pdf | technical report | DeepSeek | read |
| huggingface.co/deepseek-ai/DeepSeek-V4-Flash | model card | DeepSeek | read |
| huggingface.co/collections/deepseek-ai/deepseek-v4 | model card | DeepSeek | not fetched |
| morphllm.com/deepseek-v4-flash | third party analysis | Morph | not fetched |