Open Models Almanac
ModelsDeepSeek-V4-Flash-Vision-Exp

DeepSeek

DeepSeek-V4-Flash-Vision-Exp

What DeepSeek-V4-Flash-Vision-Exp’s own documents say it is built from — every method with the sentence that describes it.

Parameters
284B / 13B active
Context
1M
Attention
sparse (indexer)Compressed Sparse Attention + Heavily Compressed Attention with indexer, sliding window 128, attention sink; visual modules added
Experts
256 routed + 1 shared, top-6
Input
text, image
Licence
MIT
Released
2026-08-21

Against the consensus recipe

Its documents state 2 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 13 of the 19 architecture features checked — 12 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_deepseek_v4.py, configuration_deepseek_v4.py) and config.json at revision 6821d6ad36, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention code only in its codenot stated

num_key_value_heads=1 with num_attention_heads=64 gives shared-KV multi-query attention, with a single KV head broadcast via repeat_kv to…

show evidence

num_key_value_heads=1 with num_attention_heads=64 gives shared-KV multi-query attention, with a single KV head broadcast via repeat_kv to all query heads.

num_attention_heads = 64num_key_value_heads = 1
self.num_heads = config.num_attention_heads
        self.num_key_value_groups = config.num_attention_heads  # single KV head, broadcast to all
modeling_deepseek_v4.py · L789–790
self.kv_proj = nn.Linear(config.hidden_size, self.head_dim, bias=False)
modeling_deepseek_v4.py · L399
Multi-head latent attention not in its codenot stated

There is no down-projection of KV into a shared low-rank latent that is later up-projected per head; instead a single KV head (MQA) with…

show evidence

There is no down-projection of KV into a shared low-rank latent that is later up-projected per head; instead a single KV head (MQA) with q_lora_rank only compressing queries is used, so this is not DeepSeek-V2-style MLA.

q_lora_rank = 1024
self.kv_proj = nn.Linear(config.hidden_size, self.head_dim, bias=False)
        self.kv_norm = DeepseekV4RMSNorm(self.head_dim, eps=config.rms_norm_eps)
modeling_deepseek_v4.py · L801–802
Sliding-window attention code only in its codenot stated

Every attention layer applies a sliding-window KV cache of size config.sliding_window=128 in addition to any compressor branch.

show evidencesliding_window = 128
self.cumulative_length += key_states.shape[-2]
        full = torch.cat([self.keys, key_states], dim=-2)
        self.keys = full[:, :, -self.sliding_window + 1 :, :]
modeling_deepseek_v4.py · L197–199
self.sliding_window = config.sliding_window
modeling_deepseek_v4.py · L792
Interleaved sliding-window and global attention code only in its codenot stated

layer_types interleaves sliding_attention, compressed_sparse_attention, and heavily_compressed_attention layers, each with different…

show evidence

layer_types interleaves sliding_attention, compressed_sparse_attention, and heavily_compressed_attention layers, each with different full/compressed context handling, per the compress_ratios-derived layer_types.

interleave = [
                "compressed_sparse_attention" if i % 2 else "heavily_compressed_attention"
                for i in range(max(n - 2, 0))
            ]
            self.layer_types = ["heavily_compressed_attention"] * min(n, 2) + interleave
configuration_deepseek_v4.py · L270–274
self.layer_type = config.layer_types[layer_idx]
modeling_deepseek_v4.py · L785
Indexer-selected sparse attention (DeepSeek Sparse Attention) code only in its codenot stated

DeepseekV4Indexer scores compressed KV blocks per query with a lightweight weighted-ReLU scorer and keeps only the top index_topk=512…

show evidence

DeepseekV4Indexer scores compressed KV blocks per query with a lightweight weighted-ReLU scorer and keeps only the top index_topk=512 entries, used by the compressed_sparse_attention layer type.

index_topk = 512index_n_heads = 64index_head_dim = 128
scores = torch.matmul(q.float(), compressed_kv.transpose(-1, -2).float().unsqueeze(1))  # [B, S, H, T]
        scores = F.relu(scores) * self.softmax_scale
        weights = self.weights_proj(hidden_states).float() * self.weights_scaling  # [B, S, H]
modeling_deepseek_v4.py · L467–469
top_k = min(self.index_topk, compressed_len)
modeling_deepseek_v4.py · L580
Linear-attention or state-space layers alongside full attention not in its codenot stated

No linear-attention or SSM-style token mixer is implemented; all layer_types (sliding_attention, compressed_sparse_attention…

show evidence

No linear-attention or SSM-style token mixer is implemented; all layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) use the same softmax DeepseekV4Attention class.

self.self_attn = DeepseekV4Attention(config, layer_idx)
modeling_deepseek_v4.py · L1168
Gated DeltaNet layers not in its codenot stated

No gated delta-rule recurrent mixer exists anywhere in the code; all attention layers use the same softmax DeepseekV4Attention…

show the whole verdict

No gated delta-rule recurrent mixer exists anywhere in the code; all attention layers use the same softmax DeepseekV4Attention implementation.

Mamba-2 layers not in its codenot stated

No Mamba-2 selective state-space block is implemented in this file; there is no ssm_state_size or mamba head config and no SSD code path.

Learnable attention sink code only in its codenot stated

Each attention layer has a per-head learnable sink parameter concatenated into the softmax logits before normalization, gpt-oss style.

show evidence
self.sinks = nn.Parameter(torch.empty(self.num_heads))
modeling_deepseek_v4.py · L807
sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_deepseek_v4.py · L744–745
Gated attention output not in its codenot stated

The attention output is not multiplied by an input-dependent sigmoid gate before the output projection; instead it undergoes a rope…

show evidence

The attention output is not multiplied by an input-dependent sigmoid gate before the output projection; instead it undergoes a rope conjugate rotation and then the grouped output projection directly.

grouped = attn_output.reshape(*input_shape, self.config.o_groups, -1)
        grouped = self.o_a_proj(grouped).flatten(2)
        output = self.o_b_proj(grouped)
modeling_deepseek_v4.py · L881–883
QK normalization code only in its codenot stated

Queries are normalized with an unweighted RMSNorm (q_b_norm) and keys/values are normalized with kv_norm before the rotary/attention…

show evidence

Queries are normalized with an unweighted RMSNorm (q_b_norm) and keys/values are normalized with kv_norm before the rotary/attention computation.

q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)
modeling_deepseek_v4.py · L828–830
kv = self.kv_norm(self.kv_proj(hidden_states)).view(*hidden_shape).transpose(1, 2)
modeling_deepseek_v4.py · L832
Partial RoPE code only in its codenot stated

Only the trailing qk_rope_head_dim=64 of the 512-dim head carries rotary position signal, computed via partial_rotary_factor=64/512, with…

show evidence

Only the trailing qk_rope_head_dim=64 of the 512-dim head carries rotary position signal, computed via partial_rotary_factor=64/512, with the leading nope slice left untouched.

qk_rope_head_dim = 64head_dim = 512
rope_dim = cos.shape[-1]
    nope, rope = x[..., :-rope_dim], x[..., -rope_dim:]
    rotated = ((rope.float() * cos) + (rotate_half(rope).float() * sin)).to(x.dtype)
    return torch.cat([nope, rotated], dim=-1)
modeling_deepseek_v4.py · L367–370
YaRN RoPE scaling code only in its codenot stated

rope_scaling type is yarn and is applied to the compress rope-type branch (CSA/HCA layers) via ROPE_INIT_FUNCTIONS dispatch keyed by…

show evidence

rope_scaling type is yarn and is applied to the compress rope-type branch (CSA/HCA layers) via ROPE_INIT_FUNCTIONS dispatch keyed by rope_type.

rope_scaling = {"beta_fast": 32, "beta_slow": 1, "factor": 16, "original_max_position_embedd…
compress.setdefault("rope_type", "default")
            if compress["rope_type"] == "yarn":
                compress.setdefault("attention_factor", 1.0)
configuration_deepseek_v4.py · L317–319
rope_init_fn = self.compute_default_rope_parameters
            if self.rope_type[layer_type] != "default":
                rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type[layer_type]]
modeling_deepseek_v4.py · L104–106
Layers without positional encoding (NoPE) not in its codenot stated

Every attention layer applies rotary embedding (either main or compress rope type) on its trailing rope slice; there is no layer that…

show evidence

Every attention layer applies rotary embedding (either main or compress rope type) on its trailing rope slice; there is no layer that skips positional encoding entirely, only partial RoPE per head.

self.rope_layer_type = "main" if self.layer_type == "sliding_attention" else "compress"
modeling_deepseek_v4.py · L788
Mixture of experts in its codeadopted

Feed-forward MoE layers route each token to num_experts_per_tok=6 of n_routed_experts=256 expert MLPs via a learned TopKRouter (or hash…

show evidence

Feed-forward MoE layers route each token to num_experts_per_tok=6 of n_routed_experts=256 expert MLPs via a learned TopKRouter (or hash router for bootstrap layers).

n_routed_experts = 256num_experts_per_tok = 6
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
modeling_deepseek_v4.py · L1094
Shared expert code only in its codenot stated

Each MoE block adds a shared DeepseekV4MLP that every token passes through in addition to the routed experts, with n_shared_experts=1.

show evidencen_shared_experts = 1
self.shared_experts = DeepseekV4MLP(config)
modeling_deepseek_v4.py · L1137
routed = self.experts(flat, indices, weights).view(batch, seq_len, hidden_dim)
        return routed + self.shared_experts(residual)
modeling_deepseek_v4.py · L1147–1148
Auxiliary-loss-free load balancing (selection bias) code only in its codenot stated

The TopKRouter adds a per-expert e_score_correction_bias only to the scores used for top-k selection, not to the weights applied to expert…

show evidence

The TopKRouter adds a per-expert e_score_correction_bias only to the scores used for top-k selection, not to the weights applied to expert outputs, matching DeepSeek-V3's noaux_tc method.

topk_method = "noaux_tc"
self.e_score_correction_bias = nn.Buffer(torch.zeros(self.num_experts))
modeling_deepseek_v4.py · L1088
indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)
modeling_deepseek_v4.py · L1094–1095
Multi-token prediction layers code only in its codenot stated

The config declares num_nextn_predict_layers=3, a nonzero MTP module count, though the modeling file only skips their weights (ignored on…

show evidence

The config declares num_nextn_predict_layers=3, a nonzero MTP module count, though the modeling file only skips their weights (ignored on load) without instantiating them explicitly.

num_nextn_predict_layers = 3
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
modeling_deepseek_v4.py · L1266
SwiGLU feed-forward code only in its codenot stated

Both the dense MLP and the routed experts compute SiLU(gate(x)) * up(x) (clamped), consistent with hidden_act=silu.

show evidencehidden_act = "silu"
gate = self.gate_proj(x).clamp(max=self.limit)
        up = self.up_proj(x).clamp(min=-self.limit, max=self.limit)
        return self.down_proj(self.act_fn(gate) * up)
modeling_deepseek_v4.py · L1033–1035
gate, up = gate_up.chunk(2, dim=-1)
        gate = gate.clamp(max=self.limit)
        up = up.clamp(min=-self.limit, max=self.limit)
        return self.act_fn(gate) * up
modeling_deepseek_v4.py · L1073–1076

token mixer 1

softmax attention 1

DFlash attention core

“The reference inference covers the vision encoder and aligner, DFlash attention, MoE, Hyper-Connections, and the DSpark forward path.”

channel mixer 1

mixture of experts 1

Mixture of experts core

“The reference inference covers the vision encoder and aligner, DFlash attention, MoE, Hyper-Connections, and the DSpark forward path.”

normalization & residual 1

Hyper-Connections core

“The reference inference covers the vision encoder and aligner, DFlash attention, MoE, Hyper-Connections, and the DSpark forward path.”

multimodal architecture 3

Native multimodal architecture core

“It is the smallest model in our new architecture family, with native multimodal visual understanding.”

Vision encoder and multimodal aligner core

“The reference inference covers the vision encoder and aligner”

Visual modules for multimodal understanding core

“It builds on the DeepSeek-V4-Flash architecture by incorporating visual modules”

language modelling objective 1

Fill-in-the-middle completion optional

“FIM Completion(Beta)”

filed at the root 1

Post-training optimization used

“With post-training optimization, the new model achieves major improvements in tool usage and intelligent agent tasks”

mid-training & continual pretraining 1

Continued training for visual understanding core

“and undergoing continued training to unlock visual understanding capabilities.”

decoding strategy 3

Same-checkpoint target and draft weights default

“the target and draft weights therefore come from the same checkpoint.”

DSpark optional

“Enable DSpark with --speculative-algorithm DSPARK”

Chat Prefix Completion optional

“Chat Prefix Completion(Beta)”

reasoning control 1

Thinking mode optional

“The thinking modes of V4-Pro and V4-Flash now support three thinking effort levels: low / high / max.”

inference quantization 1

FP8 KV-cache quantization optional

“--kv-cache-dtype fp8”

agentic scaffolding 4

Function calling optional

“Function Calling”

Compact TXT image-path notation optional

“Both OpenAI-style JSON content blocks and the compact <image>path</image> TXT notation are supported.”

JSON mode optional

“JSON Mode”

OpenAI-style JSON content blocks optional

“Both OpenAI-style JSON content blocks and the compact <image>path</image> TXT notation are supported.”

filed at the root 1

Separation of prompt encoding from inference default

“encoding/ and inference/ deliberately remain separate: prompt formatting does not depend on PyTorch”

inference engine 1

vLLM optional

“the command below serves the model with vLLM on a single 4×GB300 node.”

evaluation harness 1

DeepSeek Harness Minimal mode used

“DeepSeek models are evaluated with the minimal mode of DeepSeek Harness as the agent framework”

unfiled 4

disk-based context caching core

“The DeepSeek API has innovatively adopted hard disk caching, reducing prices by another order of magnitude.”

DSpark forward path core

“The reference inference covers the vision encoder and aligner, DFlash attention, MoE, Hyper-Connections, and the DSpark forward path.”

hybrid reasoning architecture core

“Hybrid reasoning architecture: A single model supports both thinking mode and non-thinking mode”

cross-provider failover routing optional

“When an error occurs in an upstream provider, we can recover by routing to another healthy provider, if your request filters allow it.”

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

SourceKindPublisherStatus
api-docs.deepseek.com/updatesofficial blogDeepSeekread
huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Expmodel cardDeepSeekread
arxiv.org/abs/2606.19348technical reportDeepSeekread
openrouter.ai/deepseek/deepseek-v4-flash-vision-expvendor docsOpenRouterread
↑