Open Models Almanac
ModelsDeepSeek-V4-Flash-0731

DeepSeek · usage rank #8

DeepSeek-V4-Flash-0731

What DeepSeek-V4-Flash-0731’s own documents say it is built from — every method with the sentence that describes it.

Parameters
284B / 13B active
Context
1M
Attention
sparse (indexer)Compressed Sparse Attention + Heavily Compressed Attention with indexer, sliding window 128, attention sink
Experts
256 routed + 1 shared, top-6
Input
text
Licence
MIT
Released
2026-07-31

Rank by open-weight tokens on OpenRouter, week of 2026-10-26.

Against the consensus recipe

Its documents state 1 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 11 of the 19 architecture features checked — 11 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_deepseek_v4.py, configuration_deepseek_v4.py) and config.json at revision 7872f01b1d, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention not in its codenot stated

num_key_value_heads=1 implements shared-KV multi-query attention, but the code treats the single KV head as broadcast via repeat_kv with…

show evidence

num_key_value_heads=1 implements shared-KV multi-query attention, but the code treats the single KV head as broadcast via repeat_kv with num_key_value_groups equal to num_attention_heads, which is MQA not grouped GQA with multiple distinct KV heads—however since MQA counts as GQA per the spec, this is actually present via the single shared KV head.

num_key_value_heads = 1num_attention_heads = 64
self.num_key_value_groups = config.num_attention_heads  # single KV head, broadcast to all
modeling_deepseek_v4.py · L790
Multi-head latent attention not in its codenot stated

The attention uses a single shared KV head with low-rank query projection (q_lora_rank) but keys/values are not down-projected into a…

show evidence

The attention uses a single shared KV head with low-rank query projection (q_lora_rank) but keys/values are not down-projected into a shared low-rank latent that is up-projected per head DeepSeek-V2 style; there is no kv_lora_rank, only a direct kv_proj to head_dim.

self.kv_proj = nn.Linear(config.hidden_size, self.head_dim, bias=False)
        self.kv_norm = DeepseekV4RMSNorm(self.head_dim, eps=config.rms_norm_eps)
modeling_deepseek_v4.py · L801–802
Sliding-window attention code only in its codenot stated

Every attention layer applies a sliding-window KV cache of size sliding_window=128 regardless of layer type.

show evidencesliding_window = 128
self.cumulative_length += key_states.shape[-2]
        full = torch.cat([self.keys, key_states], dim=-2)
        self.keys = full[:, :, -self.sliding_window + 1 :, :]
modeling_deepseek_v4.py · L197–199
Interleaved sliding-window and global attention not in its codenot stated

All layers use the same sliding-window base attention (sliding_window=128 applies uniformly); the layer_types distinguish only whether a…

show evidence

All layers use the same sliding-window base attention (sliding_window=128 applies uniformly); the layer_types distinguish only whether a compressor (CSA/HCA) is added on top, not whether full global attention replaces sliding-window, so there is no interleaving of sliding vs full-context layers.

sliding_window = 128
if past_key_values is not None:  # sliding where K==V
            kv = past_key_values.update(kv, kv, self.layer_idx)[0]
modeling_deepseek_v4.py · L835–836
Indexer-selected sparse attention (DeepSeek Sparse Attention) code only in its codenot stated

The DeepseekV4Indexer scores compressed KV tokens and selects the top index_topk=512 entries per query for the CSA compressor branch…

show evidence

The DeepseekV4Indexer scores compressed KV tokens and selects the top index_topk=512 entries per query for the CSA compressor branch, implementing DeepSeek Sparse Attention's lightning indexer.

index_topk = 512index_n_heads = 64index_head_dim = 128
index_scores = index_scores.masked_fill(future_mask, float("-inf"))
            top_k_indices = index_scores.topk(top_k, dim=-1).indices  # [B, S, k]
modeling_deepseek_v4.py · L592–593
Linear-attention or state-space layers alongside full attention not in its codenot stated

No linear-attention or state-space mixer layers are implemented; all layers use the same DeepseekV4Attention softmax-attention class with…

show the whole verdict

No linear-attention or state-space mixer layers are implemented; all layers use the same DeepseekV4Attention softmax-attention class with sliding window plus optional compressors.

Gated DeltaNet layers not in its codenot stated

No gated delta-rule recurrent mixer is implemented anywhere in the modeling file.

Mamba-2 layers not in its codenot stated

No Mamba-2 selective state-space blocks are implemented in the modeling file.

Learnable attention sink code only in its codenot stated

Each attention layer has a per-head learnable sink parameter concatenated into the softmax denominator before being dropped from the output.

show evidence
sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_deepseek_v4.py · L744–745
self.sinks = nn.Parameter(torch.empty(self.num_heads))
modeling_deepseek_v4.py · L807
Gated attention output not in its codenot stated

The attention output is not multiplied by an input-dependent sigmoid gate; it goes through RoPE un-rotation and a grouped output…

show evidence

The attention output is not multiplied by an input-dependent sigmoid gate; it goes through RoPE un-rotation and a grouped output projection with no gating mechanism.

grouped = attn_output.reshape(*input_shape, self.config.o_groups, -1)
        grouped = self.o_a_proj(grouped).flatten(2)
        output = self.o_b_proj(grouped)
modeling_deepseek_v4.py · L881–883
QK normalization code only in its codenot stated

Queries are normalized with an unweighted RMSNorm per head after the q_b_proj, and keys/values share a weighted RMSNorm (kv_norm) before…

show evidence

Queries are normalized with an unweighted RMSNorm per head after the q_b_proj, and keys/values share a weighted RMSNorm (kv_norm) before the dot product.

q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)
modeling_deepseek_v4.py · L828–830
kv = self.kv_norm(self.kv_proj(hidden_states)).view(*hidden_shape).transpose(1, 2)
modeling_deepseek_v4.py · L832
Partial RoPE code only in its codenot stated

RoPE is applied only to the trailing rope_head_dim slice (64 of 512) of each query/key head, with the leading nope channels left untouched.

show evidenceqk_rope_head_dim = 64head_dim = 512
rope_dim = cos.shape[-1]
    nope, rope = x[..., :-rope_dim], x[..., -rope_dim:]
    rotated = ((rope.float() * cos) + (rotate_half(rope).float() * sin)).to(x.dtype)
    return torch.cat([nope, rotated], dim=-1)
modeling_deepseek_v4.py · L367–370
YaRN RoPE scaling code only in its codenot stated

rope_scaling specifies type yarn with factor 16 and this config's compress-branch RoPE parameters set rope_type yarn, which the rotary…

show evidence

rope_scaling specifies type yarn with factor 16 and this config's compress-branch RoPE parameters set rope_type yarn, which the rotary embedding dispatches to the YaRN init function.

rope_scaling = {"beta_fast": 32, "beta_slow": 1, "factor": 16, "original_max_position_embedd…
rope_init_fn = self.compute_default_rope_parameters
            if self.rope_type[layer_type] != "default":
                rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type[layer_type]]
modeling_deepseek_v4.py · L104–106
Layers without positional encoding (NoPE) not in its codenot stated

All attention layers apply partial RoPE to the trailing rope slice of every head; none of the softmax attention layers skip positional…

show the whole verdict

All attention layers apply partial RoPE to the trailing rope slice of every head; none of the softmax attention layers skip positional encoding entirely, so there is no NoPE/RoPE layer split.

Mixture of experts code only in its codenot stated

Each MoE layer routes tokens to num_experts_per_tok=6 of n_routed_experts=256 expert MLPs via a learned top-k router (or a hash router for…

show evidence

Each MoE layer routes tokens to num_experts_per_tok=6 of n_routed_experts=256 expert MLPs via a learned top-k router (or a hash router for the bootstrap layers).

n_routed_experts = 256num_experts_per_tok = 6
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1094–1095
Shared expert code only in its codenot stated

Each MoE block adds the output of a dense shared-expert MLP to every token's routed output, and n_shared_experts=1 in this config.

show evidencen_shared_experts = 1
routed = self.experts(flat, indices, weights).view(batch, seq_len, hidden_dim)
        return routed + self.shared_experts(residual)
modeling_deepseek_v4.py · L1147–1148
Auxiliary-loss-free load balancing (selection bias) code only in its codenot stated

The router adds a learned e_score_correction_bias to the scores only for top-k selection, not for the output weighting (which uses the raw…

show evidence

The router adds a learned e_score_correction_bias to the scores only for top-k selection, not for the output weighting (which uses the raw gathered scores), matching DeepSeek-V3's noaux_tc method set in topk_method.

topk_method = "noaux_tc"
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1094–1095
Multi-token prediction layers code only in its codenot stated

The config declares num_nextn_predict_layers=1 for MTP modules, which the loader skips but the checkpoint still carries as indicated by…

show evidence

The config declares num_nextn_predict_layers=1 for MTP modules, which the loader skips but the checkpoint still carries as indicated by the mtp key-ignore pattern.

num_nextn_predict_layers = 1
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_deepseek_v4.py · L1266
SwiGLU feed-forward code only in its codenot stated

Both the dense MLP and the expert MLPs compute SiLU(gate(x)) * up(x) with hidden_act set to silu.

show evidencehidden_act = "silu"
gate = self.gate_proj(x).clamp(max=self.limit)
        up = self.up_proj(x).clamp(min=-self.limit, max=self.limit)
        return self.down_proj(self.act_fn(gate) * up)
modeling_deepseek_v4.py · L1033–1035

multimodal architecture 1

Native multimodal architecture core

“It is the smallest model in our new architecture family, with native multimodal visual understanding.”

language modelling objective 1

Fill-in-the-middle completion optional

“FIM Completion(Beta)”

filed at the root 1

Post-training optimization used

“With post-training optimization, the new model achieves major improvements in tool usage and intelligent agent tasks”

agentic post-training 1

Re-post-training for agentic capabilities used

“a re-post-training pass focused on agentic capabilities”

decoding strategy 2

DSpark used

“DSpark speculative decoding is enabled with a single flag”

Chat Prefix Completion optional

“Chat Prefix Completion(Beta)”

reasoning control 1

Thinking mode used

“The reasoning_effort parameter now supports three levels — low, high, and max — which control how much deliberation the model spends…”

agentic scaffolding 2

Function calling optional

“Function Calling”

JSON mode optional

“JSON Mode”

1 more mentioned or evaluated
Speculative decoding mentioned

decoding strategy “it comes with a speculative decoding module attached”

1 mentioned or evaluated
A/B testing on real-world tasks mentioned

human & real-world evaluation “A/B on three real tasks — one agentic, one long-context, one knowledge.”

unfiled 6

disk-based context caching core

“The DeepSeek API has innovatively adopted hard disk caching, reducing prices by another order of magnitude.”

hybrid reasoning architecture core

“Hybrid reasoning architecture: A single model supports both thinking mode and non-thinking mode”

Failover routing to healthy upstream providers used

“When an error occurs in an upstream provider, we can recover by routing to another healthy provider, if your request filters allow it.”

OpenAI-compatible message encoding without Jinja template used

“we provide a dedicated encoding folder with Python scripts and test cases demonstrating how to encode messages in OpenAI-compatible format…”

recommended sampling parameters for deployment used

“we recommend setting the sampling parameters to temperature = 1.0, with top_p = 0.95 for agentic scenarios and top_p = 1.0 otherwise”

1 more mentioned or evaluated
cache-hit cost modeling for agent workloads mentioned

“Rule of thumb: for agent workloads, model the cache-hit price first”

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

SourceKindPublisherStatus
api-docs.deepseek.com/updatesofficial blogDeepSeekread
arxiv.org/abs/2606.19348technical reportDeepSeekread
huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731model cardDeepSeekread
huggingface.co/blog/ResterChed/deepseek-v4-flash-official-releasethird party analysisHugging Face community blogread
openrouter.ai/deepseek/deepseek-v4-flash-0731vendor docsOpenRouterread
↑