DeepSeek · usage rank #3
DeepSeek-V4.1-Flash
What DeepSeek-V4.1-Flash’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 552B / 16B active
- Context
- 1M
- Attention
- sparse (indexer)MLA + Compressed Sparse Attention 2 (Full/Reindex/Reuse modes, hierarchical indexer) + SWA, Causal Encoder-Decoder
- Experts
- 384 routed + 1 shared, top-6; plus 196B Engram memory params
- Input
- text, image
- Licence
- MIT
- Released
- 2026-09-10
Rank by open-weight tokens on OpenRouter, week of 2026-10-26.
Against the consensus recipe
Its documents state 2 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictionnot in its documents
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningcore
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modenot in its documents
- Multi-Token Predictionnot in its documents
- FP8not in its documents
Checked in its code
Its code shows 12 of the 19 architecture features checked — 11 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from vLLM (__init__.py, quant_config.py, dspark.py, mm_preprocess.py, model.py, engram.py, mhc.py, mega_mhc.py, decode.py, flashinfer_sparse.py, indexer.py, candidate_topk.py) and config.json at revision 314113f8a7, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention | not in its code | not stated | This model uses multi-head latent attention (MLA) with a shared low-rank KV latent, not grouped-query attention; num_key_value_heads=1… show evidenceThis model uses multi-head latent attention (MLA) with a shared low-rank KV latent, not grouped-query attention; num_key_value_heads=1 here reflects the MLA latent cache layout, not a GQA KV-head group. text_config.num_attention_heads = 64text_config.num_key_value_heads = 1 |
| Multi-head latent attention code only | in its code | not stated | The attention mechanism is DeepSeek-style MLA with q_lora_rank/kv-related ranks and a decoupled RoPE key dimension configured in… show evidenceThe attention mechanism is DeepSeek-style MLA with q_lora_rank/kv-related ranks and a decoupled RoPE key dimension configured in text_config. text_config.q_lora_rank = 1280text_config.qk_rope_head_dim = 64text_config.head_dim = 512 |
| Sliding-window attention code only | in its code | not stated | text_config.sliding_window is 128 and the model builds an SWA KV cache layer (swa_cache_layer) used across decoder layers and… show evidencetext_config.sliding_window is 128 and the model builds an SWA KV cache layer (swa_cache_layer) used across decoder layers and decoder-replay logic. text_config.sliding_window = 128 |
| Interleaved sliding-window and global attention code only | in its code | not stated | compress_ratios defines a per-layer pattern where some layers are SWA-only (ratio 0) and others combine SWA with a compressed/global KV… show evidencecompress_ratios defines a per-layer pattern where some layers are SWA-only (ratio 0) and others combine SWA with a compressed/global KV source (ratio 1 or 2), giving a fixed interleaving of local and global attention across the 40 layers. text_config.compress_ratios = [0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1…text_config.kv_source_layer_ids = [2, 8, 14, 20] |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) code only | in its code | not stated | An indexer (DeepseekV4Indexer / SparseAttnIndexer / SparseMQAIndexer) scores past tokens and selects index_topk=512 tokens per query… show evidenceAn indexer (DeepseekV4Indexer / SparseAttnIndexer / SparseMQAIndexer) scores past tokens and selects index_topk=512 tokens per query, configured with index_n_heads/index_head_dim, DeepSeek Sparse Attention style. text_config.index_topk = 512text_config.index_n_heads = 32text_config.index_head_dim = 128 |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | No linear-attention, SSM, Mamba, or lightning-attention token mixer is present; all layers use (sparse) softmax MLA attention, varying… show the whole verdictNo linear-attention, SSM, Mamba, or lightning-attention token mixer is present; all layers use (sparse) softmax MLA attention, varying only in compression/window pattern. |
| Gated DeltaNet layers | not in its code | not stated | No gated delta rule / delta-net recurrent layer implementation appears in the given files; the model's token mixer is MLA attention… show the whole verdictNo gated delta rule / delta-net recurrent layer implementation appears in the given files; the model's token mixer is MLA attention throughout. |
| Mamba-2 layers | not in its code | not stated | No Mamba-2 SSD state-space block implementation is present in the provided code or config. |
| Learnable attention sink code only | in its code | not stated | Attention layers pass a learned attn_sink into the sparse MLA kernels as a per-head sink logit absorbing softmax probability mass. show evidence |
| Gated attention output | code unclear | not stated | The attention/o_proj implementation (dsv41_o_proj) is imported from files not included here, so whether the attention output is multiplied… show the whole verdictThe attention/o_proj implementation (dsv41_o_proj) is imported from files not included here, so whether the attention output is multiplied by an input-dependent sigmoid gate cannot be confirmed from the given code. |
| QK normalization | code unclear | not stated | The DeepseekV4Attention base class (which would implement any q/k normalization before the dot product) is imported from a file not… show the whole verdictThe DeepseekV4Attention base class (which would implement any q/k normalization before the dot product) is imported from a file not included in the provided sources, so this cannot be confirmed. |
| Partial RoPE code only | in its code | not stated | qk_rope_head_dim (64) is smaller than head_dim (512), indicating only part of each head carries RoPE (DeepSeek-style decoupled RoPE key)… show evidenceqk_rope_head_dim (64) is smaller than head_dim (512), indicating only part of each head carries RoPE (DeepSeek-style decoupled RoPE key), consistent with MLA's partial rotary key. text_config.qk_rope_head_dim = 64text_config.head_dim = 512 |
| YaRN RoPE scaling code only | in its code | not stated | rope_scaling in text_config sets rope_type to yarn with factor 16 and associated beta parameters, and the model's rotary embedding… show evidencerope_scaling in text_config sets rope_type to yarn with factor 16 and associated beta parameters, and the model's rotary embedding (rotary_emb.cos_sin_cache) consumes this config. text_config.rope_scaling = {"rope_type": "yarn", "factor": 16, "beta_fast": 32, "beta_slow": 1, "origina… |
| Layers without positional encoding (NoPE) | code unclear | not stated | No explicit no_rope_layers/nope_layer_interval config key or code path is visible in the given files to determine whether some softmax… show the whole verdictNo explicit no_rope_layers/nope_layer_interval config key or code path is visible in the given files to determine whether some softmax attention layers skip positional encoding entirely. |
| Mixture of experts | in its code | adopted | Feed-forward layers route tokens to a subset of 384 routed experts (6 per token) via a learned router/gate, implemented in DeepseekV4MoE. show evidencetext_config.n_routed_experts = 384text_config.num_experts_per_tok = 6 |
| Shared expert code only | in its code | not stated | MoE layers include n_shared_experts=1 shared expert(s) that every token passes through, loaded/handled via shared_experts weight paths. show evidencetext_config.n_shared_experts = 1 |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | topk_method is noaux_tc (DeepSeek-V3-style) and the gate carries an e_score_correction_bias used only for expert selection, as referenced… show evidencetopk_method is noaux_tc (DeepSeek-V3-style) and the gate carries an e_score_correction_bias used only for expert selection, as referenced in weight remapping. text_config.topk_method = "noaux_tc" |
| Multi-token prediction layers code only | in its code | not stated | num_nextn_predict_layers=3 declares MTP modules, and the DSpark/MTP draft model code loads mtp.{0,1,2}.* weights from the target checkpoint. show evidencetext_config.num_nextn_predict_layers = 3 |
| SwiGLU feed-forward code only | in its code | not stated | hidden_act is silu and the MoE/MLP implementation uses gate_up_proj with w1 (gate) and w3 (up) stacking typical of SiLU-gated SwiGLU MLPs. show evidencetext_config.hidden_act = "silu" |
filed at the root 2
“we adopt a Causal Encoder-Decoder (CED) architecture, in which decoder global KV is projected from the final encoder hidden states”
“Asymmetric architecture. More intelligence, less cost.”
token mixer 7
softmax attention 1
sliding window attention 1
“Decoder SWA Bounded Replay, which only prefills the last 𝑛win tokens of the prompt for the SWA computation”
sparse attention 6
“we design Compressed Sparse Attention 2 (CSA2), which applies cross-layer reuse to global KV (including main KV and indexer K) and Top-K…”
“CSA2 has three statically assigned modes: Full, Reindex, and Reuse.”
“with sparse attention trained at a sequence length of 64K and context extended to 1M tokens at 34T tokens”
“Sparse attention is trained from scratch at a sequence length of 64K, without any dense attention warmup stages.”
sparse attention indexer 2
“the first layer assigned to Full Mode constructs a candidate pool that later re-indexing layers use as their search domain”
“CSA2 removes this overlap and absolute positional embedding. In addition, CSA2 obtains indexer K by projecting main KV entries”
channel mixer 3
mixture of experts 3
“DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model”
“It activates 8B parameters on input and 16B on output from a 552B-parameter backbone, an asymmetric split”
expert load balancing 1
“we introduce modality-specific load balancing ... for image and text tokens”
positional encoding 1
“A vision encoder (DeepSeek-ViT, trained from scratch with 2D-RoPE and 3×3 pixel-unshuffle downsampling)”
normalization & residual 2
“we upgrade the originalmHC ... design to Single-PassmHC”
“In DeepSeek-V4, we introducedmHC ... which maintains 𝑛 residual streams between adjacent Transformer blocks.”
multimodal architecture 3
“visual and text embeddings trained jointly from the start of pre-training rather than added afterward”
“we apply a pixel-unshuffle operation with 3× 3 downsampling to reduce the visual token count by a factor of nine”
“a two-layer MLP projector convert images into visual embeddings”
context capacity 1
“we integrate the Engram ... conditional memory module to strengthen model capabilities”
training stability 1
“we add a loss masking scheme that eliminates the contribution of tokens with excessive staleness”
training runtime 1
“we merge checkpoints from runs across different scaffolds or configurations, combining improvements acquired along different optimization…”
filed at the root 1
“we train DeepSeek-V4.1-Flash on a large-scale multimodal corpus comprising 45T tokens”
data sourcing 2
“public GitHub repositories that meet a star-count threshold”
“on a voluntary basis, return interaction data and feedback”
data filtering 1
“filtered to retain highly complex tasks or tasks on which model performance is poor, then deduplicated by trajectory”
synthetic data 7
“we invest in large-scale, automated pipelines for data synthesis and environment construction”
“All substantive changes lie instead in the data pipeline: large-scale automated synthesis of agent tasks and environments with progressive…”
“procedurally constructs and scales interactive agent environments in which trajectories can be collected and evaluated at low cost”
“synthesizes diverse, verifiable training tasks together with their reference solutions and reward signals”
“an agent determines whether the project can be built and fully run inside a container and whether it can be automatically verified”
“designs several sufficiently complex implementation directions, and produces concrete evaluation points, including both fail-to-pass and…”
“Using difficulty and correctness as reward signals, we iteratively train the model to construct better tasks.”
data mixture & curriculum 1
“applies rigorous filtering, deduplication, and difficulty calibration to ensure data quality and curriculum balance”
filed at the root 1
“the recipe follows the standard paradigm of supervised fine-tuning (SFT) followed by reinforcement learning (RL) and on-policy…”
reinforcement learning algorithm 3
“we use model merging to reinitialize successive RL runs”
“for samples that span multiple checkpoints, we concatenate the expert routing produced at each rollout segment rather than discarding and…”
“Responses sharing the same(𝑥,𝑏)form a subgroup, within which rewards are mean-centered to compute group-relative advantages.”
reward modelling 1
“we add the length-penalty term 𝑟 len 𝑏,𝑗 to the reward of response”
policy distillation 3
“We continuously track model capabilities and may adjust the training recipe accordingly, including the dataset mixture, per-dataset…”
“the final full-vocabulary OPD task is trained on datasets from all domains using over 40 teacher models”
“The post-training recipe follows the standard SFT → RL → on-policy distillation (OPD) paradigm without algorithmic modifications.”
rollout & RL infrastructure 15
“we extend our post-training infrastructure to allow asynchronous generation of samples”
“we partition compute nodes into multiple shards (so-called scale units)”
“Our final approach is sample-level dispatch”
“We colocate rollout and training on the same physical devices and time-share their execution”
“We use large-scale asynchronous RL in synthesized tasks to improve model performance and shape its behavior in complicated scenarios.”
“we can bound the maximum off-policy ratio, ensuring that the training data does not deviate excessively from the current model”
“we decouple agent rollout execution into an agent sandbox and a worker container”
“the dispatcher can limit concurrency on a per-dataset basis, which helps regulate the proportion of each dataset in the steady-state…”
“the persisted states are directly reused, eliminating the cost of re-prefilling and allowing interrupted samples to continue exactly where…”
“we perform sample-grained garbage collection, releasing each sample’s states as soon as it completes”
“provides a scaffold-agnostic control layer that orchestrates the rollout, normalizes heterogeneous interactions into a common trajectory…”
“rollout execution can be suspended and offloaded while preserving its full state for later resumption”
“rollout states such as KV cache and expert routing are persisted at token granularity during generation”
“we support token-level interruption: generation can be halted at any token boundary”
“we support discarding early-returned short samples to smooth the transition into the steady-state length distribution”
agentic post-training 12
“Agents communicate through a durable peer mailbox.”
“Environment construction is carried out collaboratively by multiple specialized agents.”
“If an agent crashes its environment, we treat the crash as a failed trajectory and report a "repercussion" signal to the RL framework.”
“the pipeline enables systematic replay of failures and targeted reinforcement learning against observed model weaknesses”
“incorporate them into the pipeline to generate both single-turn and multi-turn agent environments grounded in real workflows”
“multiple distinct agents attempt the task, and an independent quality-inspection agent reviews the environment together with the solving…”
“sets up dependencies, the initial working directory, test code, and task descriptions in an isolated container, performs self-testing”
“If the inspection does not pass, a repair agent fixes all identified errors, adjusts evaluation points that are too easy or too difficult…”
“Whenever a task is used in a new RL run, the resulting trajectories provide fresh evidence for quality re-auditing.”
“we construct a large set of mocked tools that reproduce the interfaces and behaviors of real-world tools and systems”
“We train Agent Team mode with an RL reward combining task performance, a collaboration bonus that encourages delegation and inter-agent…”
“Each receives a delegated task and starts in either fresh mode without lead history or fork mode with a one-time snapshot of the lead’s…”
1 more mentioned or evaluated
reinforcement learning algorithm “New pre-training methods + larger-scale RL post-training deliver benchmark results ahead of flagship models”
Not used 2
rollout & RL infrastructure “we experimented with two alternative dispatch granularities. In the first attempt, we dispatched several extra batches at the beginning…”
rollout & RL infrastructure “In the second attempt, we switched to prompt-level dispatch, where a new prompt was dispatched after one GRPO group finishes, but found…”
decoding strategy 3
“We also introduce the DSpark ... speculative decoding architecture to improve decoding efficiency through semi-autoregressive draft…”
“through semi-autoregressive draft generation and confidence-scheduled verification”
“through semi-autoregressive draft generation and confidence-scheduled verification”
reasoning control 5
“The scalar effort variable provides a deployment-time control over the cost–quality trade-off without imposing a hard token budget.”
“rlen(ℓ,b)=−min{Cmax,k(b) ℓ/Lnorm}”
“During reinforcement learning, lower effort levels apply a stronger token penalty, whereas higher effort levels permit more computation.”
“In our public API service, we expose three preset effort levels that map onto this scale:low,high, andmaxcorrespond to effort values of…”
“we prepend the following instruction to the system prompt: Reasoning Effort: {effort}”
KV cache management 2
“Compared with the previous generation, V4.1-Flash's KV cache needs just: 1/4 the HBM, 1/8 the SSD storage”
“we introduce SWA Bounded Replay, which approximately reconstructs the required SWA KV states by replaying only the most recent 𝑛win tokens”
inference quantization 1
“Combined with FP4 main KV caching (E2M1 format, one E4M3 scale per 16 channels), these designs reduce the global KV cache footprint to 890…”
inference scheduling 5
“Each replica predicts resource availability from recent measurements and makes good-enough placement decisions.”
“DSec therefore introduces a latency-sensitive (LS) execution class.”
“each node is responsible for validating the final placement decisions, enforcing a hard admission constraint”
“We apply SCHED_IDLE to non-LS tasks to minimize their scheduling priority”
“we use hardware-supported sub-NUMA partitioning and bind each worker VM to an individual NUMA domain”
inference kernel 1
“with an accompanying Mega-mHC deployment kernel that halves activation memory traffic relative to the original four-kernel implementation”
agentic scaffolding 1
“All agents share one repository checkout, making edits immediately visible to one another.”
kernel & quantization library 1
“we additionally release deepseek-recipe, a set of Rust libraries with Python bindings that provides the same prompt format as a…”
infrastructure service 3
“use core scheduling to ensure only tasks of the same priority class execute simultaneously on sibling hyperthreads to eliminate interference”
“DSec employs a custom placement engine to schedule the large number of sandboxes, by trading strong global consistency for scalability”
“DSec scales through two complementary mechanisms: sharding and scheduling with relaxed consistency.”
filed at the root 3
“Derived latency is computed by representing execution events and their collaboration dependencies as a directed acyclic graph (DAG)…”
“Scores within 0.3 of each other are considered equivalent.”
“All scaffolds use N=8 samples per task on DeepSWE v1.1 and N=3 on Terminal-Bench 2.1, with Linux containers, temperature=1.0, top_p=0.95…”
cybersecurity - vulnerability discovery; exploit development 2
“we restrict internet access and strip Git histories from the environment”
“we automatically purge transient build and package caches across diverse environments”
evaluation harness 1
“the model is evaluated with the Minimal mode of DeepSeek Harness and a 1M-token context window.”
1 more mentioned or evaluated
“For each scaffold, we keep the model checkpoint, decoding configuration, and task set identical, and only the surrounding harness ... is…”
unfiled 5
“We use per-sandbox AppArmor profiles and fine-grained eBPF-based network policies to prevent such attempts.”
“Peak/off-peak pricing continues to balance demand.”
“The encoding folder contains a self-contained Python reference implementation (encoding.py) with test cases for multi-turn conversations…”
2 more mentioned or evaluated
“we conduct preliminary experiments with DeepSeek-V4.1-Flash using DeepSeek Harness’s Agent Team mode.”
“New pre-training methods + larger-scale RL post-training deliver benchmark results ahead of flagship models”
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- DeepSeek-V4.1-Flash architecture card — LLM Architecture Gallery explainer sebastianraschka.com
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| api-docs.deepseek.com/news/news260910 | official blog | DeepSeek | read |
| huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_T… | technical report | DeepSeek | read |
| huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash | model card | DeepSeek | read |
| venturebeat.com/technology/deepseek-v4-1-flash-debuts-with-0-003-1m-off… | news | VentureBeat | not fetched |
| openrouter.ai/deepseek/deepseek-v4.1-flash | vendor docs | OpenRouter | read |
| recipes.vllm.ai/deepseek-ai/DeepSeek-V4.1-Flash | vendor docs | vLLM | not fetched |