ModelsQwen3.5-397B-A17B

Alibaba (Qwen) · released 2026-02-16

Qwen3.5-397B-A17B

What Qwen3.5-397B-A17B’s own documents say it is built from — every method with the sentence that describes it.

Curator’s note397B total / 17B active, hybrid Gated-DeltaNet linear attention plus sparse MoE, natively multimodal, 201 languages, Apache 2.0. Open-weight siblings: 122B-A10B, 35B-A3B, 27B, 9B. No base-model technical report on arXiv; the Qwen3.5-Omni report is the closest first-party architecture document.

Against the consensus recipe

Its documents state 3 of the 15 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something the documents do not say, not something the model lacks.

Data curation
- Agentic data synthesis pipelinenot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- KL alignment lossnot in its documents
Model architecture
- Mixture of Expertscore
- Hybrid Attentionnot in its documents
- Multi-Token Predictionused
Optimization
- Muonnot in its documents
- Expert Parallelismused
- Cosine decaynot in its documents
Post-training
- Group Relative Policy Optimizationnot in its documents
- Supervised fine-tuningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Configurable reasoning effortnot in its documents
- Speculative decodingnot in its documents
- FP8 KV-cache quantizationnot in its documents

Checked in its code

Its code shows 9 of the 19 architecture features checked — 6 of them are not stated in its documents.

Documents say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Feature	Code	Its documents	Evidence
Grouped-query attention code only	in its code	not stated	Full-attention layers use 32 query heads with only 2 key/value heads, repeated via repeat_kv, giving grouped-query attention. text_config.num_attention_heads = 32 text_config.num_key_value_heads = 2 `self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads` modeling_qwen3_5_moe.py · L759 `key_states = repeat_kv(key, module.num_key_value_groups) value_states = repeat_kv(value, module.num_key_value_groups)` modeling_qwen3_5_moe.py · L736–737
Multi-head latent attention	not in its code	not stated	The attention module projects K/V directly with num_key_value_heads and there is no low-rank latent kv projection or kv_lora_rank in the config or code.
Sliding-window attention	not in its code	not stated	There is no sliding_window config key or windowed mask logic in the code; full_attention layers use the standard causal mask.
Interleaved sliding-window and global attention	not in its code	not stated	The layer_types interleaving alternates linear_attention and full_attention (global), not sliding-window and global attention. text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"… `causal_mask_mapping = { "full_attention": create_causal_mask(mask_kwargs), "linear_attention": create_recurrent_attention_mask(mask_kwargs), }` modeling_qwen3_5_moe.py · L1393–1396
Indexer-selected sparse attention (DeepSeek Sparse Attention) documents disagree	not in its code	core	No indexer scoring mechanism or index_topk/index_n_heads config exists in this codebase.
Linear-attention or state-space layers alongside full attention code only	in its code	not stated	layer_types mixes linear_attention (Gated DeltaNet) layers with full_attention softmax layers in a fixed pattern (3 linear then 1 full). text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"… `if self.block_type == "linear_attention": self.linear_attn = Qwen3_5MoeGatedDeltaNet(config, layer_idx) elif self.block_type == "full_attention": self.self_attn = Qwen3_5MoeAttention(config, layer_idx)` modeling_qwen3_5_moe.py · L952–955
Gated DeltaNet layers	in its code	core	Linear-attention layers implement the gated delta rule (Qwen3_5MoeGatedDeltaNet) with a decay gate g and delta-rule chunked/recurrent update, configured via linear_num_key_heads/linear_conv_kernel_dim. text_config.linear_num_key_heads = 16 text_config.linear_conv_kernel_dim = 4 `g = -self.A_log.float().exp() * F.softplus(a.float() + self.dt_bias)` modeling_qwen3_5_moe.py · L621 `last_recurrent_state = last_recurrent_state * chunk_decay[:, :, i] + key[:, :, i].transpose(-1, -2) @ v_new` modeling_qwen3_5_moe.py · L427
Mamba-2 layers	not in its code	not stated	No Mamba-2/SSD selective state-space block or mamba_num_heads/ssm_state_size config is present; the recurrent layers are Gated DeltaNet, not Mamba-2.
Learnable attention sink	not in its code	not stated	No learned per-head sink logit is added to the attention softmax anywhere in the attention forward code.
Gated attention output	in its code	core	Attention output is multiplied by a sigmoid gate derived from a split of the q_proj output before the output projection. text_config.attn_output_gate = true `attn_output = attn_output.reshape(input_shape, -1).contiguous() attn_output = attn_output torch.sigmoid(gate)` modeling_qwen3_5_moe.py · L821–822
QK normalization code only	in its code	not stated	Queries and keys are each passed through a per-head RMSNorm (q_norm/k_norm) before RoPE and the dot product. `self.q_norm = Qwen3_5MoeRMSNorm(self.head_dim, eps=config.rms_norm_eps) # unlike olmo, only on the head dim! self.k_norm = Qwen3_5MoeRMSNorm( self.head_dim, eps=config.rms_norm_eps )` modeling_qwen3_5_moe.py · L775–778 `query_states = self.q_norm(query_states.view(hidden_shape)).transpose(1, 2) key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)` modeling_qwen3_5_moe.py · L796–797
Partial RoPE code only	in its code	not stated	partial_rotary_factor of 0.25 restricts RoPE to a quarter of the head dimension, with the rest passed through unrotated in apply_rotary_pos_emb. text_config.rope_parameters.partial_rotary_factor = 0.25 `partial_rotary_factor = config.rope_parameters.get("partial_rotary_factor", 1.0) head_dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads dim = int(head_dim * partial_rotary_factor)` modeling_qwen3_5_moe.py · L179–181 `q_rot, q_pass = q[..., :rotary_dim], q[..., rotary_dim:] k_rot, k_pass = k[..., :rotary_dim], k[..., rotary_dim:]` modeling_qwen3_5_moe.py · L701–702
YaRN RoPE scaling documents disagree	not in its code	used	rope_parameters.rope_type is set to "default", not "yarn", so no YaRN scaling is switched on for this config. text_config.rope_parameters.rope_type = "default" `self.rope_type = self.config.rope_parameters["rope_type"] rope_init_fn: Callable = self.compute_default_rope_parameters if self.rope_type != "default": rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]` modeling_qwen3_5_moe.py · L154–157
Layers without positional encoding (NoPE)	not in its code	not stated	All full_attention layers apply the same RoPE via position_embeddings; there is no no_rope_layers/nope_layer_interval config or per-layer skip of RoPE for softmax attention layers.
Mixture of experts	in its code	core	Each MoE block routes tokens via a learned TopKRouter to a subset (10 of 512) of expert MLPs. text_config.num_experts = 512 text_config.num_experts_per_tok = 10 `router_top_value, router_indices = torch.topk(router_probs, self.top_k, dim=-1) # (seq_len, top_k)` modeling_qwen3_5_moe.py · L897
Shared expert code only	in its code	not stated	A shared_expert MLP with nonzero intermediate size processes every token and is gated and added to the routed expert output. text_config.shared_expert_intermediate_size = 1024 `self.shared_expert = Qwen3_5MoeMLP(config, intermediate_size=config.shared_expert_intermediate_size) self.shared_expert_gate = torch.nn.Linear(config.hidden_size, 1, bias=False)` modeling_qwen3_5_moe.py · L909–910 `shared_expert_output = F.sigmoid(self.shared_expert_gate(hidden_states_reshaped)) * shared_expert_output expert_output = expert_output + shared_expert_output` modeling_qwen3_5_moe.py · L919–921
Auxiliary-loss-free load balancing (selection bias)	not in its code	not stated	The router only uses softmax + topk with no e_score_correction_bias or expert-selection bias term; load balancing uses the classic auxiliary loss function instead. `router_probs = torch.nn.functional.softmax(router_logits, dtype=torch.float, dim=-1) router_top_value, router_indices = torch.topk(router_probs, self.top_k, dim=-1) # (seq_len, top_k)` modeling_qwen3_5_moe.py · L896–897
Multi-token prediction layers	code unclear	used	The config declares mtp_num_hidden_layers=1 but no MTP module implementation appears in the given modeling file besides ignoring mtp.* weights on load. text_config.mtp_num_hidden_layers = 1 `_keys_to_ignore_on_load_unexpected = [r"^mtp.*"]` modeling_qwen3_5_moe.py · L1014
SwiGLU feed-forward code only	in its code	not stated	Both the dense MLP and the expert MLPs compute silu(gate_proj(x)) * up_proj(x), i.e. SwiGLU, with hidden_act set to silu. text_config.hidden_act = "silu" `def forward(self, x): down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x)) return down_proj` modeling_qwen3_5_moe.py · L839–841

Document	Kind	Publisher	Status
qwen.ai/blog?id=qwen3.5	official blog	Qwen	not fetched
huggingface.co/Qwen/Qwen3.5-397B-A17B	model card	Qwen	read
arxiv.org/abs/2604.15804	technical report	Qwen Team	read
huggingface.co/Qwen/Qwen3.5-122B-A10B	model card	Qwen	read
huggingface.co/Qwen/Qwen3.5-35B-A3B	model card	Qwen	read
github.com/QwenLM/Qwen3.5	code repo	Qwen	not fetched
recipes.vllm.ai/Qwen/Qwen3.5-397B-A17B	vendor docs	vLLM	read
openrouter.ai/qwen/qwen3.5-397b-a17b	vendor docs	OpenRouter	read
huggingface.co/blog/mlabonne/qwen35	third party analysis	Hugging Face community (mlabonne)	read
deeplearning.ai/the-batch/qwen-announces-new-open-weights-flagship-upda…	news	The Batch (DeepLearning.AI)	read

Qwen3.5-397B-A17B

Its documents state 3 of the 15 methods the field agrees on.

Data curation

Training objective

Model architecture

Optimization

Post-training

Inference & serving

Its code shows 9 of the 19 architecture features checked — 6 of them are not stated in its documents.

model architecture 28

token mixer 10

softmax attention 2

global attention 1

sliding window attention 1

sparse attention 1

linear attention & state space 4

gated delta network 3

hybrid layer stacking 3

channel mixer 5

mixture of experts 5

shared experts 2

positional encoding 3

prediction head 2

multimodal architecture 8

training objective 1

multi-token prediction objective 1

optimization 1

training parallelism 1

data curation 2

data mixture & curriculum 1

tokenization 1

post-training 11

filed at the root 2

supervised fine-tuning 1

reinforcement learning algorithm 3

preference optimization 1

policy distillation 2

rollout & RL infrastructure 1

mid-training & continual pretraining 1

inference & serving 25

decoding strategy 5

reasoning control 5

KV cache management 4

inference quantization 2

serving parallelism 2

context management 3

agentic scaffolding 4

software implementation 3

inference engine 1

agent product 1

infrastructure service 1

evaluation 2

filed at the root 1

evaluation harness 1

unfiled 25

Further reading

Sources