DeepSeek
DeepSeek-V3.2
What DeepSeek-V3.2’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 671B / 37B active
- Context
- 160K
- Attention
- sparse (indexer)MLA + DeepSeek Sparse Attention (lightning indexer)
- Experts
- 256 routed + 1 shared, top-8
- Input
- text
- Licence
- MIT
- Released
- 2025-12-01
Against the consensus recipe
Its documents state 0 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertsnot in its documents
- Multi-Token Predictionnot in its documents
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modeevaluated
- Multi-Token Predictionnot in its documents
- FP8not in its documents
Checked in its code
Its code shows 8 of the 19 architecture features checked — 7 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_deepseek_v32.py, configuration_deepseek_v32.py) and config.json at revision a7e62ac04e, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention | not in its code | not stated | num_attention_heads equals num_key_value_heads (128 each), so num_key_value_groups is 1 and there is no grouping; this model uses MLA… show evidencenum_attention_heads equals num_key_value_heads (128 each), so num_key_value_groups is 1 and there is no grouping; this model uses MLA rather than GQA. num_attention_heads = 128num_key_value_heads = 128 |
| Multi-head latent attention code only | in its code | not stated | Attention compresses KV into a shared kv_lora_rank latent with a decoupled RoPE key, then up-projects per head via kv_b_proj… show evidenceAttention compresses KV into a shared kv_lora_rank latent with a decoupled RoPE key, then up-projects per head via kv_b_proj, DeepSeek-V2/V3 style MLA. kv_lora_rank = 512q_lora_rank = 1536qk_rope_head_dim = 64 |
| Sliding-window attention | not in its code | not stated | No sliding_window config key or windowed masking logic exists anywhere in this model's attention implementation. |
| Interleaved sliding-window and global attention | not in its code | not stated | There is no layer_types or sliding_window_pattern distinguishing local vs global attention; every layer is uniformly 'indexed_attention'. show evidence |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | in its code | adopted | A DeepseekV32Indexer scores past tokens with dedicated projections and selects the top index_topk tokens per query, which are masked into… show evidenceA DeepseekV32Indexer scores past tokens with dedicated projections and selects the top index_topk tokens per query, which are masked into the attention computation (DeepSeek Sparse Attention). index_topk = 2048index_n_heads = 64index_head_dim = 128 |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | All layers use the same DeepseekV32Attention (MLA + indexer); there are no linear-attention/state-space mixer layers in this file. |
| Gated DeltaNet layers | not in its code | not stated | No gated delta-rule recurrent mixer code or related config keys (linear_num_key_heads, linear_conv_kernel_dim) are present. |
| Mamba-2 layers | not in its code | not stated | No Mamba-2/SSD selective state-space block implementation or related config keys (mamba_num_heads, ssm_state_size) exist in these files. |
| Learnable attention sink | not in its code | not stated | The attention softmax implementation (eager_attention_forward) has no extra learned sink logit term added to the denominator. show evidence |
| Gated attention output | not in its code | not stated | The attention output is reshaped and passed straight to o_proj with no sigmoid gating multiplication applied. show evidence |
| QK normalization | not in its code | not stated | Only the indexer's key gets a LayerNorm (k_norm); the main MLA query and key states (q_states, key_states) are not normalized before the… show evidenceOnly the indexer's key gets a LayerNorm (k_norm); the main MLA query and key states (q_states, key_states) are not normalized before the dot product. |
| Partial RoPE code only | in its code | not stated | Only qk_rope_head_dim=64 of each head's dimension carries RoPE while qk_nope_head_dim=128 carries no position signal, both in the main… show evidenceOnly qk_rope_head_dim=64 of each head's dimension carries RoPE while qk_nope_head_dim=128 carries no position signal, both in the main attention and the decoupled MLA key. qk_rope_head_dim = 64qk_nope_head_dim = 128 |
| YaRN RoPE scaling code only | in its code | not stated | rope_scaling.type is 'yarn' and the rotary embedding init dispatches to the YaRN ROPE_INIT_FUNCTIONS, with mscale applied to the attention… show evidencerope_scaling.type is 'yarn' and the rotary embedding init dispatches to the YaRN ROPE_INIT_FUNCTIONS, with mscale applied to the attention scaling. rope_scaling.type = "yarn"rope_scaling.factor = 40 |
| Layers without positional encoding (NoPE) | not in its code | not stated | layer_types is uniformly 'indexed_attention' for all layers and every layer applies the same partial RoPE pattern via… show evidencelayer_types is uniformly 'indexed_attention' for all layers and every layer applies the same partial RoPE pattern via apply_rotary_pos_emb_interleave; no layer fully skips positional encoding. |
| Mixture of experts code only | in its code | not stated | DeepseekV32MoE routes tokens to num_experts_per_tok of n_routed_experts experts via a learned sigmoid router with grouped top-k selection. show evidencen_routed_experts = 256num_experts_per_tok = 8 |
| Shared expert code only | in its code | not stated | DeepseekV32MoE adds a shared_experts MLP (sized by moe_intermediate_size * n_shared_experts=1) applied to every token in addition to… show evidenceDeepseekV32MoE adds a shared_experts MLP (sized by moe_intermediate_size * n_shared_experts=1) applied to every token in addition to routed experts. n_shared_experts = 1 |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | The router adds e_score_correction_bias to sigmoid scores only for top-k expert/group selection (noaux_tc method), while actual weights… show evidenceThe router adds e_score_correction_bias to sigmoid scores only for top-k expert/group selection (noaux_tc method), while actual weights come from the unbiased scores. topk_method = "noaux_tc" |
| Multi-token prediction layers | code unclear | not stated | The config declares num_nextn_predict_layers=1 but the given modeling file contains no MTP module class or usage of that key, so it is not… show evidenceThe config declares num_nextn_predict_layers=1 but the given modeling file contains no MTP module class or usage of that key, so it is not settled by these files. num_nextn_predict_layers = 1 |
| SwiGLU feed-forward code only | in its code | not stated | Both the dense DeepseekV32MLP and the MoE DeepseekV32Experts compute silu(gate(x)) * up(x), i.e. SwiGLU, matching hidden_act='silu'. show evidencehidden_act = "silu" |
token mixer 8
softmax attention 2
multi-head latent attention 2
“Therefore, we implement DSA based on the MQA (Shazeer, 2019) mode of MLA, where each latent vector (the key-value entry of MLA) will be…”
“For DeepSeek-V3.1-Terminus, the MHA mode is used for training and prefilling, while the MQA mode is used for decoding.”
sparse attention 6
“fine-grained sparse attention mechanism that reduces training and inference cost while preserving quality in long-context scenarios”
“Following indexer warm-up, we introduce the fine-grained token selection mechanism and optimize all model parameters to adapt the model to…”
sparse attention indexer 4
“The prototype of DSA primarily consists of two components: a lightning indexer and a fine-grained token selection mechanism.”
“our fine-grained token selection mechanism retrieves only the key-value entries {c_s} corresponding to the top-k index scores.”
“In this stage, we keep dense attention and freeze all model parameters except for the lightning indexer.”
“we detach the indexer input from the computational graph for separate optimization.”
channel mixer 1
mixture of experts 1
expert routing 1
“we preserve the expert routing paths used during sampling in the inference framework and enforce the same routing paths during training”
language modelling objective 1
“we imposed stricter token constraints during the training of the official DeepSeek-V3.2, aiming to optimize the trade-off between…”
auxiliary loss 2
“Based on p_t,:, we set a KL-divergence loss as the training objective of the indexer”
“In practice, we find that different domains benefit from varying strengths of KL regularization.”
data filtering 2
“we employed the DeepSeek-V32-Exp model to identify and remove samples in which the model explicitly indicated an inability or refusal to…”
“We then perform RL on this dataset using DeepSeek-V3.2 and retain only instances with non-zero pass@100”
synthetic data 9
“we developed a novel synthesis pipeline that systematically generates training data at scale. This facilitates scalable agentic…”
“To scale up agent environments and tasks in RL, we employ an automatic environment-synthesis agent that synthesizes 1,827 task-oriented…”
“The synthesis workflow primarily consists of environment and toolset construction, task synthesis, and solution generation.”
“We constructed large-scale, executable environments for software issue resolution by mining millions of issue-Pull Request (PR) pairs from…”
“Multiple answer-generation agents with heterogeneous configurations (different checkpoints, system prompts, etc.) produce diverse…”
“The agent then iteratively increases the difficulty of the task and updates the corresponding solution and verification functions.”
“We employ a multi-agent pipeline based on DeepSeek-V3.2 to generate diverse, high-quality training data.”
“A question-construction agent then explores each entity using search tools with configurable depth and breadth parameters, consolidating…”
“A verification agent with search capabilities validates all answers through multiple passes, retaining only samples where the ground-truth…”
filed at the root 1
“By implementing a robust RL protocol and scaling post-training compute, DeepSeek-V3.2 performs comparably to GPT-5.”
supervised fine-tuning 2
“We posit that the model possesses sufficient ability to accurately follow explicit instructions, thereby enabling the seamless…”
“The system prompt requires the model to output the reasoning process in the tag thinking response.”
reinforcement learning algorithm 7
“For DeepSeek-V3.2, we still adopt Group Relative Policy Optimization (GRPO) ... as the RL training algorithm.”
“As DeepSeek-V3.2-Exp, we merge reasoning, agent, and human alignment training into one RL stage.”
“we preserve the expert routing paths used during sampling in the inference framework and enforce the same routing paths during training”
“These performance gains can be attributed to the increased computational resources allocated to RL training.”
“we correct the K3 estimator (Schulman, 2020) to obtain an unbiased KL estimate using the importance-sampling ratio between the current…”
“we preserve the truncation masks during sampling from and apply them to during training, ensuring both policies share identical action…”
“we mask negative sequences that introduce significant policy divergence, as measured by the KL divergence between the data-sampling policy…”
reward modelling 4
“We then develop detailed evaluation rubrics across multiple quality dimensions and employ a generative reward model to score responses…”
“For reasoning and agent tasks, we employ rule-based outcome reward, length penalty, and language consistency reward.”
“For reasoning and agent tasks, we employ rule-based outcome reward, length penalty, and language consistency reward.”
“For reasoning and agent tasks, we employ rule-based outcome reward, length penalty, and language consistency reward.”
policy distillation 1
“Once the specialist models are prepared, they are used to produce the domain-specific data for the final checkpoint.”
agentic post-training 3
“In the IMO and CMO tasks, we employ a generate-verify-refine loop.”
“large-scale RL on synthetic data yields substantial improvements over DeepSeek-V3.2-SFT on Tau2Bench, MCP-Mark, and MCP-Universe benchmarks.”
“For tasks such as search, code engineering, and code interpretation, we employ real-world tools, including actual web search APIs, coding…”
mid-training & continual pretraining 1
“Starting from a base checkpoint of DeepSeek-V3.1-Terminus, whose context length has been extended to 128K, we perform continued…”
2 more mentioned or evaluated
agentic post-training “we conduct RL only on synthetic agentic tasks in non-thinking mode.”
agentic post-training “DeepSeek-V3.2-Exp is trained with RL only in search and code environments.”
decoding strategy 3
“Specifically, we first sampled 500 candidate solutions for each problem, then applied a multi-stage filtering pipeline.”
“we selected the 50 samples with the longest thinking traces for final submission.”
“Employing these strategies in RL training is also advantageous, as it avoids sampling extremely low-probability tokens that would be used…”
reasoning control 2
“In the IMO and CMO tasks, we employ a generate-verify-refine loop.”
“Users can control the reasoning behaviour with the reasoning enabled boolean.”
context management 3
“To address this, we employ a context management method to derive the final score.”
“Historical reasoning content is discarded only when a new user message is introduced to the conversation.”
“we introduce context management employing simple strategies to extend token budgets at test time”
agentic scaffolding 4
“DeepSeek-V3.2 introduces significant updates to its chat template compared to prior versions. The primary changes involve a revised format…”
“a straightforward strategy for integrating these two capabilities is through carefully designed prompting.”
“We utilize Jupyter Notebook as a code interpreter to address complex reasoning tasks.”
“Important: ALWAYS adhere to this exact format for tool use: {TOOLCALL-FORMAT}”
7 more mentioned or evaluated
reasoning control “The performance of non-thinking mode is slightly worse than the thinking mode, but still competitive.”
context management “Discard-all, which resets the context by discarding all previous tool call history”
context management “Discard-75%, which discards the first 75% tool call history in the trajectory to free up spaces”
reasoning control “Parallel-fewest-step, which samples N independent trajectories and selects the trajectory with the fewest steps.”
reasoning control “test-time compute can be scaled either serially through context management or in parallel”
context management “Summary, which summarizes the overflowed trajectory and re-initiates the rollout”
reasoning control “test-time compute can be scaled either serially through context management or in parallel”
filed at the root 2
“we eliminated invalid submissions that failed to pass the provided sample test cases or exceeded the length constraints”
“Tool-use benchmarks are evaluated using the standard function call format”
benchmark 1
“An environment is deemed successfully built only when applying the gold patch results in a non-zero count of false-to-positive (F2P) test…”
unfiled 2
“we can recover by routing to another healthy provider”
“From the remaining valid candidates, we selected the 50 samples with the longest thinking traces for final submission.”
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- DeepSeek-V3.2 architecture card — LLM Architecture Gallery explainer sebastianraschka.com
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| arxiv.org/abs/2512.02556 | technical report | DeepSeek | read |
| huggingface.co/deepseek-ai/DeepSeek-V3.2 | model card | DeepSeek | read |
| huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/main/assets/paper.pdf | technical report | DeepSeek | read |
| openrouter.ai/deepseek/deepseek-v3.2 | vendor docs | OpenRouter | read |
| lmsys.org/blog/2025-05-05-large-scale-ep | vendor docs | LMSYS | not fetched |