DeepSeek · usage rank #8
DeepSeek-V4-Flash-0731
What DeepSeek-V4-Flash-0731’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 284B / 13B active
- Context
- 1M
- Attention
- sparse (indexer)Compressed Sparse Attention + Heavily Compressed Attention with indexer, sliding window 128, attention sink
- Experts
- 256 routed + 1 shared, top-6
- Input
- text
- Licence
- MIT
- Released
- 2026-07-31
Rank by open-weight tokens on OpenRouter, week of 2026-10-26.
Against the consensus recipe
Its documents state 1 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertsnot in its documents
- Multi-Token Predictionnot in its documents
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modeused
- Multi-Token Predictionnot in its documents
- FP8not in its documents
Checked in its code
Its code shows 11 of the 19 architecture features checked — 11 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_deepseek_v4.py, configuration_deepseek_v4.py) and config.json at revision 7872f01b1d, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention | not in its code | not stated | num_key_value_heads=1 implements shared-KV multi-query attention, but the code treats the single KV head as broadcast via repeat_kv with… show evidencenum_key_value_heads=1 implements shared-KV multi-query attention, but the code treats the single KV head as broadcast via repeat_kv with num_key_value_groups equal to num_attention_heads, which is MQA not grouped GQA with multiple distinct KV heads—however since MQA counts as GQA per the spec, this is actually present via the single shared KV head. num_key_value_heads = 1num_attention_heads = 64 |
| Multi-head latent attention | not in its code | not stated | The attention uses a single shared KV head with low-rank query projection (q_lora_rank) but keys/values are not down-projected into a… show evidenceThe attention uses a single shared KV head with low-rank query projection (q_lora_rank) but keys/values are not down-projected into a shared low-rank latent that is up-projected per head DeepSeek-V2 style; there is no kv_lora_rank, only a direct kv_proj to head_dim. |
| Sliding-window attention code only | in its code | not stated | Every attention layer applies a sliding-window KV cache of size sliding_window=128 regardless of layer type. show evidencesliding_window = 128 |
| Interleaved sliding-window and global attention | not in its code | not stated | All layers use the same sliding-window base attention (sliding_window=128 applies uniformly); the layer_types distinguish only whether a… show evidenceAll layers use the same sliding-window base attention (sliding_window=128 applies uniformly); the layer_types distinguish only whether a compressor (CSA/HCA) is added on top, not whether full global attention replaces sliding-window, so there is no interleaving of sliding vs full-context layers. sliding_window = 128 |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) code only | in its code | not stated | The DeepseekV4Indexer scores compressed KV tokens and selects the top index_topk=512 entries per query for the CSA compressor branch… show evidenceThe DeepseekV4Indexer scores compressed KV tokens and selects the top index_topk=512 entries per query for the CSA compressor branch, implementing DeepSeek Sparse Attention's lightning indexer. index_topk = 512index_n_heads = 64index_head_dim = 128 |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | No linear-attention or state-space mixer layers are implemented; all layers use the same DeepseekV4Attention softmax-attention class with… show the whole verdictNo linear-attention or state-space mixer layers are implemented; all layers use the same DeepseekV4Attention softmax-attention class with sliding window plus optional compressors. |
| Gated DeltaNet layers | not in its code | not stated | No gated delta-rule recurrent mixer is implemented anywhere in the modeling file. |
| Mamba-2 layers | not in its code | not stated | No Mamba-2 selective state-space blocks are implemented in the modeling file. |
| Learnable attention sink code only | in its code | not stated | Each attention layer has a per-head learnable sink parameter concatenated into the softmax denominator before being dropped from the output. show evidence |
| Gated attention output | not in its code | not stated | The attention output is not multiplied by an input-dependent sigmoid gate; it goes through RoPE un-rotation and a grouped output… show evidenceThe attention output is not multiplied by an input-dependent sigmoid gate; it goes through RoPE un-rotation and a grouped output projection with no gating mechanism. |
| QK normalization code only | in its code | not stated | Queries are normalized with an unweighted RMSNorm per head after the q_b_proj, and keys/values share a weighted RMSNorm (kv_norm) before… show evidenceQueries are normalized with an unweighted RMSNorm per head after the q_b_proj, and keys/values share a weighted RMSNorm (kv_norm) before the dot product. |
| Partial RoPE code only | in its code | not stated | RoPE is applied only to the trailing rope_head_dim slice (64 of 512) of each query/key head, with the leading nope channels left untouched. show evidenceqk_rope_head_dim = 64head_dim = 512 |
| YaRN RoPE scaling code only | in its code | not stated | rope_scaling specifies type yarn with factor 16 and this config's compress-branch RoPE parameters set rope_type yarn, which the rotary… show evidencerope_scaling specifies type yarn with factor 16 and this config's compress-branch RoPE parameters set rope_type yarn, which the rotary embedding dispatches to the YaRN init function. rope_scaling = {"beta_fast": 32, "beta_slow": 1, "factor": 16, "original_max_position_embedd… |
| Layers without positional encoding (NoPE) | not in its code | not stated | All attention layers apply partial RoPE to the trailing rope slice of every head; none of the softmax attention layers skip positional… show the whole verdictAll attention layers apply partial RoPE to the trailing rope slice of every head; none of the softmax attention layers skip positional encoding entirely, so there is no NoPE/RoPE layer split. |
| Mixture of experts code only | in its code | not stated | Each MoE layer routes tokens to num_experts_per_tok=6 of n_routed_experts=256 expert MLPs via a learned top-k router (or a hash router for… show evidenceEach MoE layer routes tokens to num_experts_per_tok=6 of n_routed_experts=256 expert MLPs via a learned top-k router (or a hash router for the bootstrap layers). n_routed_experts = 256num_experts_per_tok = 6 |
| Shared expert code only | in its code | not stated | Each MoE block adds the output of a dense shared-expert MLP to every token's routed output, and n_shared_experts=1 in this config. show evidencen_shared_experts = 1 |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | The router adds a learned e_score_correction_bias to the scores only for top-k selection, not for the output weighting (which uses the raw… show evidenceThe router adds a learned e_score_correction_bias to the scores only for top-k selection, not for the output weighting (which uses the raw gathered scores), matching DeepSeek-V3's noaux_tc method set in topk_method. topk_method = "noaux_tc" |
| Multi-token prediction layers code only | in its code | not stated | The config declares num_nextn_predict_layers=1 for MTP modules, which the loader skips but the checkpoint still carries as indicated by… show evidenceThe config declares num_nextn_predict_layers=1 for MTP modules, which the loader skips but the checkpoint still carries as indicated by the mtp key-ignore pattern. num_nextn_predict_layers = 1 |
| SwiGLU feed-forward code only | in its code | not stated | Both the dense MLP and the expert MLPs compute SiLU(gate(x)) * up(x) with hidden_act set to silu. show evidencehidden_act = "silu" |
multimodal architecture 1
“It is the smallest model in our new architecture family, with native multimodal visual understanding.”
language modelling objective 1
“FIM Completion(Beta)”
filed at the root 1
“With post-training optimization, the new model achieves major improvements in tool usage and intelligent agent tasks”
agentic post-training 1
“a re-post-training pass focused on agentic capabilities”
decoding strategy 2
“DSpark speculative decoding is enabled with a single flag”
“Chat Prefix Completion(Beta)”
reasoning control 1
“The reasoning_effort parameter now supports three levels — low, high, and max — which control how much deliberation the model spends…”
agentic scaffolding 2
“Function Calling”
“JSON Mode”
1 more mentioned or evaluated
decoding strategy “it comes with a speculative decoding module attached”
1 mentioned or evaluated
human & real-world evaluation “A/B on three real tasks — one agentic, one long-context, one knowledge.”
unfiled 6
“The DeepSeek API has innovatively adopted hard disk caching, reducing prices by another order of magnitude.”
“Hybrid reasoning architecture: A single model supports both thinking mode and non-thinking mode”
“When an error occurs in an upstream provider, we can recover by routing to another healthy provider, if your request filters allow it.”
“we provide a dedicated encoding folder with Python scripts and test cases demonstrating how to encode messages in OpenAI-compatible format…”
“we recommend setting the sampling parameters to temperature = 1.0, with top_p = 0.95 for agentic scenarios and top_p = 1.0 otherwise”
1 more mentioned or evaluated
“Rule of thumb: for agent workloads, model the cache-hit price first”
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| api-docs.deepseek.com/updates | official blog | DeepSeek | read |
| arxiv.org/abs/2606.19348 | technical report | DeepSeek | read |
| huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 | model card | DeepSeek | read |
| huggingface.co/blog/ResterChed/deepseek-v4-flash-official-release | third party analysis | Hugging Face community blog | read |
| openrouter.ai/deepseek/deepseek-v4-flash-0731 | vendor docs | OpenRouter | read |