DeepSeek-V4-Flash-0731

What DeepSeek-V4-Flash-0731’s own documents say it is built from — every method with the sentence that describes it.

Curator’s noteThe official release of DeepSeek-V4-Flash, superseding the April preview: same architecture and size (284B total / 13B active; 304B on the hub including the attached DSpark speculative-decoding module), with the gains coming entirely from a rebuilt post-training pipeline aimed at agentic work. 1M context, MIT. Shares the arXiv 2606.19348 report with V4-Flash and V4-Pro — fetched once, credited to all of them.

Rank by open-weight tokens on OpenRouter, week of 2026-09-21.

Against the consensus recipe

Its documents state 1 of the 15 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something the documents do not say, not something the model lacks.

Data curation
- Agentic data synthesis pipelinenot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- KL alignment lossnot in its documents
Model architecture
- Mixture of Expertsnot in its documents
- Hybrid Attentionnot in its documents
- Multi-Token Predictionnot in its documents
Optimization
- Muonnot in its documents
- Expert Parallelismnot in its documents
- Cosine decaynot in its documents
Post-training
- Group Relative Policy Optimizationnot in its documents
- Supervised fine-tuningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Configurable reasoning effortused
- Speculative decodingmentioned
- FP8 KV-cache quantizationnot in its documents

Checked in its code

Its code shows 13 of the 19 architecture features checked — 13 of them are not stated in its documents.

Documents say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_deepseek_v4.py, configuration_deepseek_v4.py) and config.json at revision 7872f01b1d, by anthropic/claude-sonnet-5 on 2026-09-25.

Feature	Code	Its documents	Evidence
Grouped-query attention code only	in its code	not stated	num_key_value_heads=1 gives a single shared KV head (multi-query attention) broadcast to all 64 query heads via repeat_kv/num_key_value_groups. num_key_value_heads = 1 num_attention_heads = 64 `self.num_key_value_groups = config.num_attention_heads # single KV head, broadcast to all` modeling_deepseek_v4.py · L782
Multi-head latent attention	not in its code	not stated	Attention uses a shared single KV head projected directly to head_dim (MQA) with q/o low-rank projections, not a DeepSeek-V2-style down/up-projected latent KV cache; kv_lora_rank is absent from config. `self.kv_proj = nn.Linear(config.hidden_size, self.head_dim, bias=False) self.kv_norm = DeepseekV4RMSNorm(self.head_dim, eps=config.rms_norm_eps)` modeling_deepseek_v4.py · L793–794
Sliding-window attention code only	in its code	not stated	sliding_window=128 is used by all layer types' cache update (DeepseekV4HCACache/base sliding window) and passed into attention as the local window. sliding_window = 128 `self.sliding_window = config.sliding_window` modeling_deepseek_v4.py · L784
Interleaved sliding-window and global attention code only	in its code	not stated	layer_types interleaves sliding_attention, compressed_sparse_attention, and heavily_compressed_attention layers, with only sliding_attention layers being pure local-window and the others extending attention to compressed long-range context. `self.layer_type = config.layer_types[layer_idx] # Sliding-only layers use the "main" (plain θ=10000) rope; CSA/HCA layers # share the same yarn-scaled "compress" rope as their compressor.` modeling_deepseek_v4.py · L777–779 `interleave = [ "compressed_sparse_attention" if i % 2 else "heavily_compressed_attention" for i in range(max(n - 2, 0)) ] self.layer_types = ["heavily_compressed_attention"] * min(n, 2) + interleave` configuration_deepseek_v4.py · L271–275
Indexer-selected sparse attention (DeepSeek Sparse Attention) code only	in its code	not stated	DeepseekV4Indexer scores compressed KV entries and returns top index_topk=512 indices per query, used by CSA layers to build a sparse block_bias restricting attention. index_topk = 512 index_n_heads = 64 index_head_dim = 128 `top_k_indices = index_scores.topk(top_k, dim=-1).indices # [B, S, k] invalid = top_k_indices >= causal_threshold.unsqueeze(-1) return torch.where(invalid, torch.full_like(top_k_indices, -1), top_k_indices)` modeling_deepseek_v4.py · L585–587
Linear-attention or state-space layers alongside full attention	not in its code	not stated	All layer types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) are softmax-attention variants using DeepseekV4Attention; there is no linear-attention/SSM mixer layer type in the code.
Gated DeltaNet layers	not in its code	not stated	No gated delta-rule recurrent module exists in the code; all layers use DeepseekV4Attention softmax attention.
Mamba-2 layers	not in its code	not stated	No Mamba-2/SSD selective state-space block is implemented; the model is built entirely from DeepseekV4Attention layers.
Learnable attention sink code only	in its code	not stated	Each attention layer has a learnable per-head sink parameter concatenated to the logits before softmax, gpt-oss style. `sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1) combined_logits = torch.cat([attn_weights, sinks], dim=-1)` modeling_deepseek_v4.py · L736–737 `self.sinks = nn.Parameter(torch.empty(self.num_heads))` modeling_deepseek_v4.py · L799
Gated attention output	not in its code	not stated	The attention output is only rotated back and projected through the grouped output projection; there is no sigmoid gate multiplying the attention output before o_proj.
QK normalization code only	in its code	not stated	Queries are normalized with an unweighted RMSNorm per head after q_b_proj, and keys/values are normalized with kv_norm before the dot product. `q = self.q_b_proj(q_residual).view(hidden_shape).transpose(1, 2) q = self.q_b_norm(q) q = apply_rotary_pos_emb(q, cos, sin)` modeling_deepseek_v4.py · L820–822 `kv = self.kv_norm(self.kv_proj(hidden_states)).view(hidden_shape).transpose(1, 2)` modeling_deepseek_v4.py · L824
Partial RoPE code only	in its code	not stated	Only a trailing rope_head_dim slice (qk_rope_head_dim=64 out of head_dim=512) of each head receives rotary embeddings, the rest (nope) is left untouched. qk_rope_head_dim = 64 head_dim = 512 `rope_dim = cos.shape[-1] nope, rope = x[..., :-rope_dim], x[..., -rope_dim:] rotated = ((rope.float() * cos) + (rotate_half(rope).float() * sin)).to(x.dtype) return torch.cat([nope, rotated], dim=-1)` modeling_deepseek_v4.py · L359–362
YaRN RoPE scaling code only	in its code	not stated	rope_scaling type is yarn and is applied to the compress rope-type branch used by CSA/HCA layers via ROPE_INIT_FUNCTIONS lookup keyed by rope_type. rope_scaling = {"beta_fast": 32, "beta_slow": 1, "factor": 16, "original_max_position_embedd… `rope_init_fn = self.compute_default_rope_parameters if self.rope_type[layer_type] != "default": rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type[layer_type]]` modeling_deepseek_v4.py · L105–107
Layers without positional encoding (NoPE)	not in its code	not stated	All attention layers apply partial RoPE to the trailing rope slice of every head (either 'main' or 'compress' rope type); no layer type is fully without positional encoding. `cos, sin = position_embeddings[self.rope_layer_type]` modeling_deepseek_v4.py · L817
Mixture of experts code only	in its code	not stated	Feed-forward layers use DeepseekV4SparseMoeBlock with a learned TopKRouter (or hash router) selecting num_experts_per_tok=6 of n_routed_experts=256 experts per token. n_routed_experts = 256 num_experts_per_tok = 6 `indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices weights = scores.gather(1, indices)` modeling_deepseek_v4.py · L1086–1087
Shared expert code only	in its code	not stated	Each MoE block includes a DeepseekV4MLP shared_experts module (n_shared_experts=1) added unconditionally to the routed output. n_shared_experts = 1 `self.shared_experts = DeepseekV4MLP(config)` modeling_deepseek_v4.py · L1129 `routed = self.experts(flat, indices, weights).view(batch, seq_len, hidden_dim) return routed + self.shared_experts(residual)` modeling_deepseek_v4.py · L1139–1140
Auxiliary-loss-free load balancing (selection bias) code only	in its code	not stated	The router adds e_score_correction_bias to the scores only for top-k selection (topk_method noaux_tc), while the weights used for output combination come from the unbiased scores. topk_method = "noaux_tc" `indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices weights = scores.gather(1, indices)` modeling_deepseek_v4.py · L1086–1087
Multi-token prediction layers code only	in its code	not stated	Config declares num_nextn_predict_layers=1 and the modeling code explicitly ignores unexpected MTP weight keys on load, indicating the checkpoint carries an MTP module the code skips. num_nextn_predict_layers = 1 `_keys_to_ignore_on_load_unexpected = [r"(^\|\.)mtp\..*"]` modeling_deepseek_v4.py · L1259
SwiGLU feed-forward code only	in its code	not stated	Both the dense MLP and the MoE experts compute SiLU(gate(x)) * up(x) with hidden_act=silu. hidden_act = "silu" `gate = self.gate_proj(x).clamp(max=self.limit) up = self.up_proj(x).clamp(min=-self.limit, max=self.limit) return self.down_proj(self.act_fn(gate) * up)` modeling_deepseek_v4.py · L1025–1027

evaluation 1

human & real-world evaluation 1

A/B testing on real tasks mentioned

unfiled 6

disk-based context caching core

hybrid reasoning architecture core

Failover routing to healthy upstream providers used

OpenAI-compatible message encoding without Jinja template used

cache-hit cost modeling for agent workloads mentioned

Sources

The curated document list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

Document	Kind	Publisher	Status
api-docs.deepseek.com/updates	official blog	DeepSeek	read
arxiv.org/abs/2606.19348	technical report	DeepSeek	read
huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731	model card	DeepSeek	read
huggingface.co/blog/ResterChed/deepseek-v4-flash-official-release	third party analysis	Hugging Face community blog	read
openrouter.ai/deepseek/deepseek-v4-flash-0731	vendor docs	OpenRouter	read

DeepSeek-V4-Flash-0731

Its documents state 1 of the 15 methods the field agrees on.

Data curation

Training objective

Model architecture

Optimization

Post-training

Inference & serving

Its code shows 13 of the 19 architecture features checked — 13 of them are not stated in its documents.

model architecture 1

multimodal architecture 1

training objective 1

language modelling objective 1

post-training 2

filed at the root 1

agentic post-training 1

inference & serving 6

decoding strategy 3

reasoning control 1

agentic scaffolding 2

evaluation 1

human & real-world evaluation 1

unfiled 6

Sources