DeepSeek
DeepSeek-V4-Flash-Vision-Exp
What DeepSeek-V4-Flash-Vision-Exp’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 284B / 13B active
- Context
- 1M
- Attention
- sparse (indexer)Compressed Sparse Attention + Heavily Compressed Attention with indexer, sliding window 128, attention sink; visual modules added
- Experts
- 256 routed + 1 shared, top-6
- Input
- text, image
- Licence
- MIT
- Released
- 2026-08-21
Against the consensus recipe
Its documents state 2 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictionnot in its documents
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modeoptional
- Multi-Token Predictionnot in its documents
- FP8not in its documents
Checked in its code
Its code shows 13 of the 19 architecture features checked — 12 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_deepseek_v4.py, configuration_deepseek_v4.py) and config.json at revision 6821d6ad36, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention code only | in its code | not stated | num_key_value_heads=1 with num_attention_heads=64 gives shared-KV multi-query attention, with a single KV head broadcast via repeat_kv to… show evidencenum_key_value_heads=1 with num_attention_heads=64 gives shared-KV multi-query attention, with a single KV head broadcast via repeat_kv to all query heads. num_attention_heads = 64num_key_value_heads = 1 |
| Multi-head latent attention | not in its code | not stated | There is no down-projection of KV into a shared low-rank latent that is later up-projected per head; instead a single KV head (MQA) with… show evidenceThere is no down-projection of KV into a shared low-rank latent that is later up-projected per head; instead a single KV head (MQA) with q_lora_rank only compressing queries is used, so this is not DeepSeek-V2-style MLA. q_lora_rank = 1024 |
| Sliding-window attention code only | in its code | not stated | Every attention layer applies a sliding-window KV cache of size config.sliding_window=128 in addition to any compressor branch. show evidencesliding_window = 128 |
| Interleaved sliding-window and global attention code only | in its code | not stated | layer_types interleaves sliding_attention, compressed_sparse_attention, and heavily_compressed_attention layers, each with different… show evidencelayer_types interleaves sliding_attention, compressed_sparse_attention, and heavily_compressed_attention layers, each with different full/compressed context handling, per the compress_ratios-derived layer_types. |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) code only | in its code | not stated | DeepseekV4Indexer scores compressed KV blocks per query with a lightweight weighted-ReLU scorer and keeps only the top index_topk=512… show evidenceDeepseekV4Indexer scores compressed KV blocks per query with a lightweight weighted-ReLU scorer and keeps only the top index_topk=512 entries, used by the compressed_sparse_attention layer type. index_topk = 512index_n_heads = 64index_head_dim = 128 |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | No linear-attention or SSM-style token mixer is implemented; all layer_types (sliding_attention, compressed_sparse_attention… show evidenceNo linear-attention or SSM-style token mixer is implemented; all layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) use the same softmax DeepseekV4Attention class. |
| Gated DeltaNet layers | not in its code | not stated | No gated delta-rule recurrent mixer exists anywhere in the code; all attention layers use the same softmax DeepseekV4Attention… show the whole verdictNo gated delta-rule recurrent mixer exists anywhere in the code; all attention layers use the same softmax DeepseekV4Attention implementation. |
| Mamba-2 layers | not in its code | not stated | No Mamba-2 selective state-space block is implemented in this file; there is no ssm_state_size or mamba head config and no SSD code path. |
| Learnable attention sink code only | in its code | not stated | Each attention layer has a per-head learnable sink parameter concatenated into the softmax logits before normalization, gpt-oss style. show evidence |
| Gated attention output | not in its code | not stated | The attention output is not multiplied by an input-dependent sigmoid gate before the output projection; instead it undergoes a rope… show evidenceThe attention output is not multiplied by an input-dependent sigmoid gate before the output projection; instead it undergoes a rope conjugate rotation and then the grouped output projection directly. |
| QK normalization code only | in its code | not stated | Queries are normalized with an unweighted RMSNorm (q_b_norm) and keys/values are normalized with kv_norm before the rotary/attention… show evidenceQueries are normalized with an unweighted RMSNorm (q_b_norm) and keys/values are normalized with kv_norm before the rotary/attention computation. |
| Partial RoPE code only | in its code | not stated | Only the trailing qk_rope_head_dim=64 of the 512-dim head carries rotary position signal, computed via partial_rotary_factor=64/512, with… show evidenceOnly the trailing qk_rope_head_dim=64 of the 512-dim head carries rotary position signal, computed via partial_rotary_factor=64/512, with the leading nope slice left untouched. qk_rope_head_dim = 64head_dim = 512 |
| YaRN RoPE scaling code only | in its code | not stated | rope_scaling type is yarn and is applied to the compress rope-type branch (CSA/HCA layers) via ROPE_INIT_FUNCTIONS dispatch keyed by… show evidencerope_scaling type is yarn and is applied to the compress rope-type branch (CSA/HCA layers) via ROPE_INIT_FUNCTIONS dispatch keyed by rope_type. rope_scaling = {"beta_fast": 32, "beta_slow": 1, "factor": 16, "original_max_position_embedd… |
| Layers without positional encoding (NoPE) | not in its code | not stated | Every attention layer applies rotary embedding (either main or compress rope type) on its trailing rope slice; there is no layer that… show evidenceEvery attention layer applies rotary embedding (either main or compress rope type) on its trailing rope slice; there is no layer that skips positional encoding entirely, only partial RoPE per head. |
| Mixture of experts | in its code | adopted | Feed-forward MoE layers route each token to num_experts_per_tok=6 of n_routed_experts=256 expert MLPs via a learned TopKRouter (or hash… show evidenceFeed-forward MoE layers route each token to num_experts_per_tok=6 of n_routed_experts=256 expert MLPs via a learned TopKRouter (or hash router for bootstrap layers). n_routed_experts = 256num_experts_per_tok = 6 |
| Shared expert code only | in its code | not stated | Each MoE block adds a shared DeepseekV4MLP that every token passes through in addition to the routed experts, with n_shared_experts=1. show evidencen_shared_experts = 1 |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | The TopKRouter adds a per-expert e_score_correction_bias only to the scores used for top-k selection, not to the weights applied to expert… show evidenceThe TopKRouter adds a per-expert e_score_correction_bias only to the scores used for top-k selection, not to the weights applied to expert outputs, matching DeepSeek-V3's noaux_tc method. topk_method = "noaux_tc" |
| Multi-token prediction layers code only | in its code | not stated | The config declares num_nextn_predict_layers=3, a nonzero MTP module count, though the modeling file only skips their weights (ignored on… show evidenceThe config declares num_nextn_predict_layers=3, a nonzero MTP module count, though the modeling file only skips their weights (ignored on load) without instantiating them explicitly. num_nextn_predict_layers = 3 |
| SwiGLU feed-forward code only | in its code | not stated | Both the dense MLP and the routed experts compute SiLU(gate(x)) * up(x) (clamped), consistent with hidden_act=silu. show evidencehidden_act = "silu" |
token mixer 1
softmax attention 1
“The reference inference covers the vision encoder and aligner, DFlash attention, MoE, Hyper-Connections, and the DSpark forward path.”
channel mixer 1
mixture of experts 1
“The reference inference covers the vision encoder and aligner, DFlash attention, MoE, Hyper-Connections, and the DSpark forward path.”
normalization & residual 1
“The reference inference covers the vision encoder and aligner, DFlash attention, MoE, Hyper-Connections, and the DSpark forward path.”
multimodal architecture 3
“It is the smallest model in our new architecture family, with native multimodal visual understanding.”
“The reference inference covers the vision encoder and aligner”
“It builds on the DeepSeek-V4-Flash architecture by incorporating visual modules”
language modelling objective 1
“FIM Completion(Beta)”
filed at the root 1
“With post-training optimization, the new model achieves major improvements in tool usage and intelligent agent tasks”
mid-training & continual pretraining 1
“and undergoing continued training to unlock visual understanding capabilities.”
decoding strategy 3
“the target and draft weights therefore come from the same checkpoint.”
“Enable DSpark with --speculative-algorithm DSPARK”
“Chat Prefix Completion(Beta)”
reasoning control 1
“The thinking modes of V4-Pro and V4-Flash now support three thinking effort levels: low / high / max.”
inference quantization 1
“--kv-cache-dtype fp8”
agentic scaffolding 4
“Function Calling”
“Both OpenAI-style JSON content blocks and the compact <image>path</image> TXT notation are supported.”
“JSON Mode”
“Both OpenAI-style JSON content blocks and the compact <image>path</image> TXT notation are supported.”
filed at the root 1
“encoding/ and inference/ deliberately remain separate: prompt formatting does not depend on PyTorch”
inference engine 1
“the command below serves the model with vLLM on a single 4×GB300 node.”
evaluation harness 1
“DeepSeek models are evaluated with the minimal mode of DeepSeek Harness as the agent framework”
unfiled 4
“The DeepSeek API has innovatively adopted hard disk caching, reducing prices by another order of magnitude.”
“The reference inference covers the vision encoder and aligner, DFlash attention, MoE, Hyper-Connections, and the DSpark forward path.”
“Hybrid reasoning architecture: A single model supports both thinking mode and non-thinking mode”
“When an error occurs in an upstream provider, we can recover by routing to another healthy provider, if your request filters allow it.”
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| api-docs.deepseek.com/updates | official blog | DeepSeek | read |
| huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp | model card | DeepSeek | read |
| arxiv.org/abs/2606.19348 | technical report | DeepSeek | read |
| openrouter.ai/deepseek/deepseek-v4-flash-vision-exp | vendor docs | OpenRouter | read |