Model techniques map
ModelsDeepSeek-V4.1-Flash

DeepSeek · usage rank #2 · released 2026-09-10

DeepSeek-V4.1-Flash

What DeepSeek-V4.1-Flash’s own documents say it is built from — every method with the sentence that describes it.

OpenRouterHugging Face

Curator’s note552B backbone + 196B Engram conditional-memory parameters, and the first DeepSeek built on a Causal Encoder-Decoder (CED) stack: 20 causal-encoder layers feeding 20 decoder layers, whose global KV cache is projected from the final encoder states — 8B parameters activated per token at prefill, 16B at decode. Compressed Sparse Attention 2 (per-layer Full / Reindex / Reuse modes + a hierarchical sparse indexer), FP4 main KV cache and SWA Bounded Replay bring the global cache to 890 bytes/token, ~1/4 of V4-Flash. Also Single-Pass mHC, DSpark speculative decoding and a trained-from-scratch DeepSeek-ViT; 45T multimodal pre-training tokens, 1M context, MIT. Its own technical report ships in the HF repo, not on arXiv.

Rank by open-weight tokens on OpenRouter, week of 2026-09-21.

Against the consensus recipe

Its documents state 4 of the 15 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something the documents do not say, not something the model lacks.

  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 13 of the 19 architecture features checked — 8 of them are not stated in its documents.

Documents say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from vLLM (__init__.py, quant_config.py, dspark.py, mm_preprocess.py, model.py, engram.py, engram.py, mhc.py, mega_mhc.py, model_state.py, vl_model.py, vl_cudagraph.py, attention.py, compressor.py, sparse_mla.py) and config.json at revision dba1be0a40, by anthropic/claude-sonnet-5 on 2026-09-25.

FeatureCodeIts documentsEvidence
Grouped-query attention not in its codenot stated

This model uses multi-head latent attention (MLA) with a single-head latent KV cache rather than GQA; num_key_value_heads=1 reflects the MLA latent, not shared KV head groups.

text_config.num_key_value_heads = 1
self.n_heads = config.num_attention_heads
        assert self.n_heads % tp_size == 0
        self.n_local_heads = self.n_heads // tp_size
        self.q_lora_rank = config.q_lora_rank
attention.py · L282–285
Multi-head latent attention code only in its codenot stated

Keys/values are compressed into a shared low-rank latent (kv_norm/compressor) and queries/keys use a decoupled RoPE dimension, DeepSeek-V2/V3 style MLA.

text_config.q_lora_rank = 1280text_config.qk_rope_head_dim = 64
self.q_lora_rank = config.q_lora_rank
        self.o_lora_rank = config.o_lora_rank
        self.head_dim = config.head_dim
        self.rope_head_dim = config.qk_rope_head_dim
        self.nope_head_dim = self.head_dim - self.rope_head_dim
attention.py · L285–289
Sliding-window attention in its code core

Every layer maintains a sliding-window KV cache (DeepseekV4SWACache) sized from config.sliding_window=128, and some layers (compress_ratio=0) are pure sliding-window attention.

text_config.sliding_window = 128
self.window_size = config.sliding_window
attention.py · L292
Interleaved sliding-window and global attention code only in its codenot stated

The compress_ratios array fixes, per layer, whether it is pure sliding-window (0), full compressed (1), or ratio-2 compressed with sparse attention over the full context (2), interleaving sliding-window-only layers with full/compressed-context layers.

text_config.compress_ratios = [0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1…
compress_ratios = getattr(config, "compress_ratios", None)
        if compress_ratios is not None and layer_id < len(compress_ratios):
            self.compress_ratio = int(compress_ratios[layer_id])
        else:
            # MTP layers past the configured list are pure sliding-window.
            self.compress_ratio = 0
attention.py · L300–305
Indexer-selected sparse attention (DeepSeek Sparse Attention) code only in its codenot stated

A DeepseekV4Indexer with dedicated heads/head_dim scores past (compressed) tokens and each query attends only to the indexer's selected top-k tokens (index_topk), matching the DeepSeek Sparse Attention / lightning indexer pattern.

text_config.index_topk = 512text_config.index_n_heads = 32text_config.index_head_dim = 128
self.topk_tokens = config.index_topk
        self.n_head = config.index_n_heads  # 32
        self.head_dim = config.index_head_dim  # 128
attention.py · L1245–1247
Linear-attention or state-space layers alongside full attention not in its codenot stated

No linear-attention or SSM token-mixing layers are implemented; every decoder layer uses the same DeepseekV4Attention (sparse/sliding-window MLA) mechanism.

Gated DeltaNet layers not in its codenot stated

No gated delta-rule recurrent mixer code exists in the given files; attention layers are exclusively sparse/sliding-window MLA.

Mamba-2 layers not in its codenot stated

No Mamba-2/SSD selective state-space implementation is present in the given files.

Learnable attention sink code only in its codenot stated

Each attention layer has a learned per-head attn_sink parameter added as an extra logit in the sparse-MLA attention softmax denominator (supports_sink is True for the backend).

self.attn_sink = nn.Parameter(
            torch.full((self.padded_heads,), -float("inf"), dtype=torch.float32),
            requires_grad=False,
        )
attention.py · L349–352
    @classmethod
    def supports_sink(cls) -> bool:
        return True
sparse_mla.py · L119–121
Gated attention output not in its codenot stated

No sigmoid gate is applied to the attention output before the o_proj; the Engram module has a sigmoid gate but that is a separate n-gram injection mechanism, not attention-output gating.

QK normalization code only in its codenot stated

Queries and keys (the fused q-lora / kv projection) are RMSNorm'd (q_norm, kv_norm) before the attention dot product.

self.q_norm = RMSNorm(self.q_lora_rank, self.eps)
attention.py · L362
self.kv_norm = RMSNorm(self.head_dim, self.eps)
attention.py · L372
Partial RoPE code only in its codenot stated

Only qk_rope_head_dim (64) of the 512-dim head carries RoPE (nope_head_dim = head_dim - rope_head_dim), a decoupled/partial RoPE key as in MLA.

text_config.qk_rope_head_dim = 64text_config.head_dim = 512
self.rope_head_dim = config.qk_rope_head_dim
        self.nope_head_dim = self.head_dim - self.rope_head_dim
attention.py · L288–289
YaRN RoPE scaling code only in its codenot stated

rope_scaling in the config sets rope_type=yarn with factor/beta_fast/beta_slow, and build_deepseek_v4_rope constructs the rotary embedding from this config for use in attention.

text_config.rope_scaling = {"rope_type": "yarn", "factor": 16, "beta_fast": 32, "beta_slow": 1, "origina…
self.rotary_emb = build_deepseek_v4_rope(
            config,
            head_dim=self.head_dim,
            rope_head_dim=self.rope_head_dim,
            max_position_embeddings=config.max_position_embeddings,
            compress_ratio=self.compress_ratio,
        )
attention.py · L399–405
Layers without positional encoding (NoPE) not in its codenot stated

There is no per-layer flag disabling RoPE entirely for some softmax attention layers; all layers apply RoPE to the rotary portion of Q/K (partial RoPE), and no no_rope_layers/nope_layer_interval mechanism is present.

Mixture of experts in its code core

Feed-forward layers (DeepseekV4MoE) route each token to a learned top-k subset of n_routed_experts routed experts.

text_config.n_routed_experts = 384text_config.num_experts_per_tok = 6
n_routed_experts = config.n_routed_experts
        n_activated_experts = config.num_experts_per_tok
model.py · L118–119
Shared expert in its code used

Each MoE layer also has n_shared_experts=1 shared expert(s) that all tokens pass through in addition to the routed experts.

text_config.n_shared_experts = 1
def _pad_shared_expert_weight(
        quant_config: QuantizationConfig | None,
        name: str,
        loaded_weight: torch.Tensor,
    ) -> torch.Tensor:
model.py · L1051–1055
Auxiliary-loss-free load balancing (selection bias) in its code core

topk_method is 'noaux_tc' (DeepSeek-V3 aux-loss-free scheme) and the router's e_score_correction_bias is loaded and mapped from the gate's bias in the weight loader.

text_config.topk_method = "noaux_tc"
".ffn.gate.bias": ".ffn.gate.e_score_correction_bias",
model.py · L1179
Multi-token prediction layers code only in its codenot stated

The config declares num_nextn_predict_layers=3 and the checkpoint's mtp.{0,1,2}.* weights are loaded as extra next-token-prediction decoder layers (DSpark draft model reuses this count too).

text_config.num_nextn_predict_layers = 3
self.num_dspark_layers = (
            getattr(config, "n_mtp_layers", None)
            or getattr(config, "num_nextn_predict_layers", None)
            or 3
        )
dspark.py · L87–91
SwiGLU feed-forward in its code used

hidden_act is 'silu' and MoE/shared expert MLPs use a gate_up_proj (SiLU-gated) * up projection pattern typical of SwiGLU feed-forward blocks.

text_config.hidden_act = "silu"
("gate_up_proj", "w1", 0),
            ("gate_up_proj", "w3", 1),
model.py · L926–927

model architecture 43

filed at the root 2

Causal Encoder-Decoder (CED) architecture core
Asymmetric architecture core

token mixer 15

Cross-layer attention sharing core
Micro-batch-level shared-state lifetime management used

softmax attention 3

channel mixer 11

dense feed-forward network 2

SwiGLU used
SwiGLU clamping used

positional encoding 1

2D rotary position embedding used

normalization & residual 2

Single-Pass mHC core
RMSNorm used

multimodal architecture 10

3×3 pixel-unshuffle downsampling core
Joint vision-language pre-training core
Multimodal mixture-of-experts Transformer core
Vision encoder–MLP projector pathway core
Disaggregated encoder training used
Discarding the LLM after vision-encoder training used
High-resolution autoregressive vision-encoder fine-tuning used
Linear-projection patch embedding used
Staged vision-encoder freezing used
Two-layer MLP vision projector used

context capacity 2

Engram core
Row-wise distributed partitioning of embedding tables used

training objective 4

filed at the root 1

SigLIP sigmoid contrastive loss used

language modelling objective 1

Next-token prediction used

auxiliary loss 2

Sample-level attention masking used
Sequence-level balance loss used

optimization 13

filed at the root 1

Momentum update with Sinkhorn balancing core

optimizer 4

Per-Head Muon default
Sinkhorn-balanced update default
Muon used
AdamW used

learning-rate schedule 2

Engram learning-rate scaling used
Linear warmup followed by cosine decay used

training stability 1

Loss masking for excessively stale tokens used

training parallelism 3

Communication-Computation Overlap core
Load-Balanced Image Sharding used
Pipeline Payload Extensions used

training runtime 2

Offline checkpoint merging used
Batch-level embedding prefetch used

data curation 30

data sourcing 2

Image-code pairs and computer-use trajectories used
Star-threshold-filtered GitHub repositories used

data filtering 5

Coding-agent session filtering and trajectory deduplication used
Filtering low-information model-generated content used
Heuristic and statistical filtering with deduplication and quality models used
Image-text relevance filtering used
SmolVLM-based image-text quality scoring used

deduplication 2

Joint prefetching and assignment of training samples used
Semantic image deduplication used

synthetic data 14

Automated data synthesis and environment-construction pipelines core
Procedural construction and scaling of interactive agent environments core
Progressively scaled synthesis of agent tasks and environments core
Synthesis of verifiable tasks with solutions and reward signals core
Automated batch synthesis of RL training data used
Container buildability and verifiability check used
Containerized coding-environment construction with self-testing and trace removal used
Failure-case and negative-feedback-driven environment generation used
Mocked tools for agent environments used
Multi-agent collaborative environment construction used
Multi-agent task-attempt validation used
Recycling filtered documents into image-text pairs used
Reward-driven task-construction training used
Task formalization as a problem–environment–verification triplet used

data mixture & curriculum 5

Filtering, deduplication, and difficulty calibration core
Deterministic pre-splitting of ultra-long documents used
Domain-specific multimodal datasets used
Scaling-ladder-guided data and model planning used
Union-based integration of text-only and multimodal corpora used

sequence packing 2

Best-fit packing used
Interleaved image-text sequences used

post-training 41

filed at the root 1

SFT followed by RL and on-policy distillation default

supervised fine-tuning 1

Autoregressive fine-tuning used

reinforcement learning algorithm 10

Effort-Conditioned Reinforcement Learning core
Concatenated Routing Replay used
Derived-Latency Penalty used
Effort-Dependent Length Penalty used
Exponentially Decaying Token-Penalty Coefficient used
Failure Replay and Targeted Reinforcement Learning used
Group-Relative Advantage Normalization Within Effort-Level Subgroups used
Progressive Scaling of RL Data, Tasks, and Rollouts used
RL Reward with Collaboration Bonus and Derived-Latency Penalty used
Reinforcement Learning Post-Training mentioned

reward modelling 1

Collaboration bonus used

policy distillation 3

On-Policy Distillation used
Large-Scale On-Policy Distillation used
SFT–RL–On-Policy Distillation Pipeline used

rollout & RL infrastructure 23

Asynchronous reinforcement learning framework core
Sample-level dispatch default
Co-located RL training used
Asynchronous sample generation used
Bounding the maximum off-policy ratio used
Compute-node sharding into scale units used
Consistent configuration transitions used
Decoupling agent rollout into sandbox and worker container used
Discarding early-returned short samples used
Dynamic training recipe reconfiguration used
Full-lifecycle monitoring of RL tasks used
Incremental image transfer used
Large-scale asynchronous RL in synthesized tasks used
Per-dataset rollout concurrency limiting used
Per-task upper bound on in-flight rollout samples used
Preemption-safe suspension and offloading of rollout execution used
Sample-grained garbage collection used
Scaffold-agnostic rollout control layer used
Scaling RL along training compute and number of scaffolds used
Token-granularity persistence of rollout states used
Token-level interruption used
Batch-level rollout dispatch not used
Prompt-level dispatch not used

agentic post-training 2

Repair-agent environment correction loop used
Repercussion signal for agent-crashed environments used

inference & serving 40

decoding strategy 3

DSpark core
Throughput-aware dynamic verification-length scheduling core
Speculative decoding used

reasoning control 6

Configurable reasoning effort core
Capped linear reasoning-token length deduction core
Effort-dependent exponential token-penalty schedule core
Deployment-time scalar effort control default
Test-time compute scaling evaluated
Reward-optimized preferred reasoning length evaluated

KV cache management 8

Compressed KV caching core
SWA Bounded Replay core
Cross-layer KV-cache reuse core
Encoder SWA bounded replay used
Least-recently-used eviction used
Persistent KV-cache management used
Zero SWA caching not used
Exact SWA KV reconstruction via full multi-layer replay not used

inference quantization 2

FP4 KV-cache quantization core
Post-RoPE KV-cache quantization default

serving parallelism 1

Encoder-Prefill-Decode disaggregation used

inference scheduling 3

Latency-sensitive execution class with priority isolation used
Per-node hard admission constraint used
Sub-NUMA partitioning with per-VM NUMA binding used

inference kernel 4

Kernel fusion core
Fused RoPE-attention-RoPE-cast kernel used
Mega-mHC used
Single-Pass mHC used

agentic scaffolding 13

Multi-agent collaboration used
Agent Team mode used
App-server mode with adapted tool schemas used
AppArmor and eBPF sandbox policies used
Asynchronous teammate spawning used
Fresh and fork teammate initialization modes used
Lead-agent interruption of teammates used
Mandatory per-turn tool calls with submission marker used
Native task delegation used
Programmatic TypeScript tool calling used
Shared task board with revision checks used
Single-Bash-tool scaffold interface used
Web search extension used

software implementation 5

kernel & quantization library 1

deepseek-recipe optional

agent product 1

Claude Agent SDK with version-specific native tool interfaces used

infrastructure service 3

DeepSeek Elastic Compute (DSec) core
Custom placement engine used
Durable peer mailbox used

evaluation 15

filed at the root 7

Almost@1 used
Deadline-bounded rollout evaluation used
Evaluation equivalence threshold used
Mean@5 used
Standardized agent evaluation protocol used
Model–harness co-design optional
Single-agent and multi-agent evaluation configurations evaluated

benchmark 3

Fail-to-pass and pass-to-pass evaluation points used
High-confidence ProgramBench subset used
Multiple-rollout evaluation used

evaluation harness 2

DeepSeek Harness Minimal mode used
Cross-scaffold evaluation used

judge 1

Independent quality-inspection agent used

human & real-world evaluation 2

Reward hacking mitigation in evaluations used
Systematic robustness-boundary characterization used

other 1

filed at the root 1

Joint Optimization of Architecture, Cache Precision, and Deployment core

unfiled 9

Blockwise candidate-pool selection for sparse indexing core
contrastive pre-training of vision encoder used
low-resolution contrastive pre-training of vision encoder used
Peak/off-peak pricing used
replicated eventual-consistency placement decisions based on measured resource availability used
shadow replicas for shared indexers in pipeline-parallel training used
sharding and relaxed-consistency scheduling for horizontal scaling used
encoding.py reference prompt-encoding implementation optional
Pre-training methods mentioned

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated document list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

DocumentKindPublisherStatus
api-docs.deepseek.com/news/news260910official blogDeepSeekread
huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_T…technical reportDeepSeekread
huggingface.co/deepseek-ai/DeepSeek-V4.1-Flashmodel cardDeepSeekread
venturebeat.com/technology/deepseek-v4-1-flash-debuts-with-0-003-1m-off…newsVentureBeatnot fetched
openrouter.ai/deepseek/deepseek-v4.1-flashvendor docsOpenRouterread