Alibaba (Qwen)
Qwen3.5-397B-A17B
What Qwen3.5-397B-A17B’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 397B / 17B active
- Context
- 256K
- Attention
- linear hybridGated DeltaNet linear layers + gated full GQA layers
- Experts
- 512 routed + 1 shared, top-10
- Input
- text, image, video
- Licence
- Apache 2.0
- Released
- 2026-02-16
Against the consensus recipe
Its documents state 7 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionused
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictionused
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismused
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modedefault
- Multi-Token Predictionoptional
- FP8used
Checked in its code
Its code shows 10 of the 19 architecture features checked — 6 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_qwen3_5_moe.py, configuration_qwen3_5_moe.py) and config.json at revision 8472618112, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention code only | in its code | not stated | Full-attention layers project 32 query heads down to 2 KV heads, which are repeated via repeat_kv to match the query heads. show evidencetext_config.num_attention_heads = 32text_config.num_key_value_heads = 2 |
| Multi-head latent attention | not in its code | not stated | No kv_lora_rank/q_lora_rank-based latent projection exists; attention uses standard per-head q/k/v projections with GQA instead. |
| Sliding-window attention | not in its code | not stated | There is no sliding_window config key or windowed-mask code; attention is either full or linear (recurrent), not local-window. |
| Interleaved sliding-window and global attention | not in its code | not stated | layer_types interleaves full_attention and linear_attention (not sliding-window) layers, so this is a linear/full hybrid, not… show evidencelayer_types interleaves full_attention and linear_attention (not sliding-window) layers, so this is a linear/full hybrid, not sliding-window/full. text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"… |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) documents disagree | not in its code | adopted | No index_topk / indexer module is present in the code or config. |
| Linear-attention or state-space layers alongside full attention code only | in its code | not stated | layer_types mixes linear_attention (Gated DeltaNet) layers with full_attention softmax layers in a fixed pattern (every 4th layer is full… show evidencelayer_types mixes linear_attention (Gated DeltaNet) layers with full_attention softmax layers in a fixed pattern (every 4th layer is full attention). text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"…text_config.full_attention_interval = 4 |
| Gated DeltaNet layers | in its code | adopted | Qwen3_5MoeGatedDeltaNet implements the gated delta rule with a decay gate (A_log/dt_bias) and delta-rule state update, used in… show evidenceQwen3_5MoeGatedDeltaNet implements the gated delta rule with a decay gate (A_log/dt_bias) and delta-rule state update, used in linear_attention layers. text_config.linear_num_key_heads = 16text_config.linear_conv_kernel_dim = 4 |
| Mamba-2 layers | not in its code | not stated | The linear layers implement Gated DeltaNet, not Mamba-2 SSD blocks; there is no ssm_state_size or mamba_num_heads config, and no… show the whole verdictThe linear layers implement Gated DeltaNet, not Mamba-2 SSD blocks; there is no ssm_state_size or mamba_num_heads config, and no selective-SSM code. |
| Learnable attention sink | not in its code | not stated | No learned per-head sink logit or extra softmax denominator column appears in the attention code. |
| Gated attention output | in its code | adopted | Attention output is multiplied by a sigmoid gate derived from a chunked q_proj output before the output projection. show evidencetext_config.attn_output_gate = true |
| QK normalization code only | in its code | not stated | Queries and keys in full_attention layers are normalized per head with RMSNorm (q_norm/k_norm) before RoPE and the dot product. show evidence |
| Partial RoPE code only | in its code | not stated | partial_rotary_factor of 0.25 restricts RoPE to a quarter of the head dimension, with the remainder passed through unrotated in… show evidencepartial_rotary_factor of 0.25 restricts RoPE to a quarter of the head dimension, with the remainder passed through unrotated in apply_rotary_pos_emb. text_config.rope_parameters = {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "defa… |
| YaRN RoPE scaling documents disagree | not in its code | adopted | rope_parameters.rope_type is "default", not "yarn", so no YaRN scaling is switched on. |
| Layers without positional encoding (NoPE) | not in its code | not stated | All full_attention layers apply rotary embeddings via position_embeddings/apply_rotary_pos_emb; there is no no_rope_layers config or… show the whole verdictAll full_attention layers apply rotary embeddings via position_embeddings/apply_rotary_pos_emb; there is no no_rope_layers config or per-layer skip of RoPE among attention layers. |
| Mixture of experts | in its code | adopted | Qwen3_5MoeSparseMoeBlock routes tokens to a top-k subset of 512 experts via a learned router (Qwen3_5MoeTopKRouter). show evidencetext_config.num_experts = 512text_config.num_experts_per_tok = 10 |
| Shared expert code only | in its code | not stated | Each MoE block also includes a shared_expert MLP (sized by shared_expert_intermediate_size=1024) applied to every token and gated, added… show evidenceEach MoE block also includes a shared_expert MLP (sized by shared_expert_intermediate_size=1024) applied to every token and gated, added to the routed-expert output. text_config.shared_expert_intermediate_size = 1024 |
| Auxiliary-loss-free load balancing (selection bias) | not in its code | not stated | The router (Qwen3_5MoeTopKRouter) computes softmax probabilities and top-k directly with no e_score_correction_bias or selection-only bias… show evidenceThe router (Qwen3_5MoeTopKRouter) computes softmax probabilities and top-k directly with no e_score_correction_bias or selection-only bias added to logits. |
| Multi-token prediction layers | in its code | adopted | The text config declares mtp_num_hidden_layers=1, indicating an extra next-next-token prediction module count greater than zero (weights… show evidenceThe text config declares mtp_num_hidden_layers=1, indicating an extra next-next-token prediction module count greater than zero (weights are ignored on load per _keys_to_ignore_on_load_unexpected = [r"^mtp.*"]). text_config.mtp_num_hidden_layers = 1 |
| SwiGLU feed-forward code only | in its code | not stated | Both the dense MLP and expert MLPs compute act_fn(gate_proj(x)) * up_proj(x) with hidden_act=silu, i.e. SwiGLU. show evidencetext_config.hidden_act = "silu" |
token mixer 8
softmax attention 2
global attention 1
“MiniMax-M2.5 is the only full attention model (see this article) with Multi-Head Attention (MHA) for reliability purposes.”
sliding window attention 1
“The dynamic attention window size training mechanism is adopted for guaranting balance performance of inference under real-time prefill…”
sparse attention 1
“DeepSeek Sparse Attention”
linear attention & state space 4
“integrates a linear attention mechanism with a sparse mixture-of-experts model”
gated delta network 3
“Gated DeltaNet:- Number of Linear Attention Heads: 64 for V and 16 for QK”
“Gated Delta Networks combined with sparse Mixture-of-Experts deliver high-throughput inference with minimal latency and cost overhead.”
“Number of Attention Heads: 32 for Q and 2 for KV”
hybrid layer stacking 1
“Gated Delta Networks combined with sparse Mixture-of-Experts deliver high-throughput inference with minimal latency and cost overhead.”
channel mixer 5
mixture of experts 5
“integrates a linear attention mechanism with a sparse mixture-of-experts model”
“activates only 17 billion of its 397 billion total parameters per forward pass”
“The overall backbone adopts a Hybrid Mixture-of-Experts (MoE) design, improving scalability while better balancing capacity and efficiency…”
positional encoding 6
“All the notable open-source frameworks implement static YaRN, which means the scaling factor remains constant regardless of input length”
“YaRN is currently supported by several inference frameworks, e.g., transformers, vllm, ktransformers and sglang.”
“we recommend using RoPE scaling techniques to handle long texts effectively, e.g., YaRN.”
“It is also recommended to modify the factor as needed. For example, if the typical context length for your application is 524,288 tokens…”
“we recommend using RoPE scaling techniques to handle long texts effectively”
“Modifying the model configuration file”
prediction head 2
“MTP: trained with multi-steps”
“To model the residual codebooks, it employs a multi-token prediction (MTP) module, which enables fine-grained modeling and control of…”
multimodal architecture 8
“Early fusion training on multimodal tokens achieves cross-generational parity with Qwen3 and outperforms Qwen3-VL models across reasoning…”
“Qwen3.5-Omni builds on the Thinker–Talker architecture introduced in Qwen2.5-Omni (Xu et al., 2025a)”
“we resample the waveform to 16 kHz and convert it into a 128-channel mel-spectrogram using a 25 ms window and a 10 ms hop size.”
“We use transformer based audio encoder trained from scratch in attention-encoder-decoder model AuT, as is shown in Figure 3.”
“the generated multi-codebook tokens are converted to waveform by a causal and streaming ConvNet codec decoder.”
“we prepend each video or audio-video temporal patch with an explicit timestamp represented as a formatted text string in seconds, allowing…”
“Qwen3.5-Omni adopts the RVQ-based speech representation introduced in Qwen3-Omni (Xu et al., 2025b), which substantially improves…”
“It is recommended to set the longest_edge parameter in the video_preprocessor_config file to 469,762,048 (corresponding to 224k video…”
Not used 1
positional encoding “we apply TM-RoPE to endow the model with temporal awareness. However, we find that directly encoding absolute time through temporal…”
multi-token prediction objective 1
“MTP: trained with multi-steps”
training parallelism 1
“use --language-model-only to skip loading the vision encoder and free up memory for KV cache, and enable Expert Parallelism.”
data mixture & curriculum 1
“The pre-training of Qwen3.5-Omni is structured into three distinct stages.”
tokenization 1
“we use the Qwen3.5 tokenizer (Team, 2026), which adopts byte-level byte-pair encoding with a vocabulary size of 250k (up from 150k)”
filed at the root 2
“Qwen3.5 was trained with reinforcement learning scaled across what the team describes as "million-agent environments with progressively…”
“The post-training phase employs a three-stage strategy for the Thinker, designed to preserve the model’s capabilities across all…”
supervised fine-tuning 1
“we perform lightweight speaker fine-tuning on top of the base model, enabling Qwen3.5-Omni to faithfully capture target speaker…”
reinforcement learning algorithm 3
“we incorporate rule-based rewards and adopt GSPO (Zheng et al., 2025) to further improve overall capability and training stability across…”
“we introduce Interaction-Aligned RL, a third-stage reinforcement learning procedure aimed at optimizing the model for interaction quality.”
“progressively complex task distributions for robust real-world adaptability”
preference optimization 1
“We further align model behavior with human preferences through Direct Preference Optimization (DPO) (Rafailov et al., 2023).”
policy distillation 2
“we introduce a second-stage training procedure based on on-policy distillation (OPD), with the goal of distilling the model’s stronger…”
“we first train a suite of domain-specialized teacher models via independent Supervised Fine-Tuning (SFT) and reinforcement learning (RL).”
rollout & RL infrastructure 1
“asynchronous RL frameworks supporting massive-scale agent scaffolds and environment orchestration.”
mid-training & continual pretraining 1
“we perform data quality stratification through a dedicated curation pipeline and conduct continual pre-training (CPT) on high-quality…”
decoding strategy 5
“For supported frameworks, you can adjust the presence_penaltyparameter between 0 and 2 to reduce endless repetitions.”
“For latency-sensitive workloads at low concurrency, enable MTP-1 speculative decoding and disable prefix caching.”
“--speculative-algo NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4”
“--speculative-algo NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4”
“We recommend using the following set of sampling parameters for generation”
reasoning control 5
“Qwen3.5 models operate in thinking mode by default, generating thinking content signified by thinking... before producing the final…”
“Qwen3.5 models operate in thinking mode by default, generating thinking content”
“Add --reasoning-parser qwen3 --default-chat-template-kwargs '{"enable_thinking": false}'to disable reasoning mode via command-line…”
“We suggest using the following sets of sampling parameters depending on the mode and task type:”
“We recommend using an output length of 32,768 tokens for most queries. For benchmarking on highly complex problems, such as those found in…”
KV cache management 4
“we retain the chunked-prefilling mechanism as implemented in Qwen3-Omni and Qwen2.5-Omni, whose audio and vision encoders are capable of…”
“average price customers are actually paying after prompt caching”
“use --language-model-only to skip loading the vision encoder and free up memory for KV cache, and enable Expert Parallelism.”
“enable MTP-1 speculative decoding and disable prefix caching.”
inference quantization 2
“We recommend using the official FP8 checkpoint Qwen/Qwen3.5-397B-A17B-FP8 for optimal serving efficiency.”
“We recommend using the NVFP4 checkpoint nvidia/Qwen3.5-397B-A17B-NVFP4 for optimal serving efficiency on GB200 nodes.”
serving parallelism 2
“For multimodal workloads, use --mm-encoder-tp-mode data for data-parallel vision encoding and --mm-processor-cache-type shm for…”
“using tensor parallel on 8 GPUs”
context management 2
“In multi-turn conversations, the historical model output should only include the final output part and does not need to include the…”
“most search agents built on our model adopt a simple context-folding strategy(256k): once the cumulative Tool Response length reaches a…”
agentic scaffolding 3
“we introduce a dedicated system prompt for Talker that specifies target voice characteristics, thereby enabling both zero-shot voice…”
“To define the available tools, you can use the MCP configuration file, use the integrated tool of Qwen-Agent, or integrate other tools by…”
“To enable tool calling, add --enable-auto-tool-choice --tool-call-parser qwen3_coder to the serve command.”
1 more mentioned or evaluated
context management “using the same discard-all strategy as DeepSeek-V3.2 and Kimi K2.5 achieved 78.6”
Not used 1
reasoning control “Qwen3.5 does not officially support the soft switch of Qwen3, i.e., /thinkand /nothink.”
inference engine 1
“--language-model-only”
agent product 1
“We recommend using Qwen-Agent to quickly build Agent applications with Qwen3.5.”
infrastructure service 1
“automatically retries on the next-best provider when one returns an error”
filed at the root 1
“We recommend using prompts to standardize model outputs when benchmarking.”
evaluation harness 1
“Please show your choice in the answerfield with only the choice letter”
unfiled 25
“Gated Attention:- Number of Attention Heads: 32 for Q and 2 for KV”
“ARIA enforces an adaptive rate constraint: for any prefix of the generated sequence, the cumulative speech-to-text token ratio must not…”
“Reinforcement learning scaled across million-agent environments with progressively complex task distributions for robust real-world…”
“By default, `fps=2` and `do_sample_frames=True`.”
“OpenRouter's API is OpenAI-compatible — most SDKs work by just swapping the base URL.”
“We recommend using an output length of 32,768 tokens for most queries.”
“We recommend using an output length of 32,768 tokens for most queries.”
“Qwen3.5-Omni also supports cross-lingual voice cloning, where the model is required to preserve speaker identity while generating speech…”
“the model is able to transfer the target speaker characteristics to all 29 evaluated languages while maintaining stable generation quality”
“We employ a four-stage training pipeline for Talker, enabling Qwen3.5-Omni to generate natural and contextually appropriate spoken…”
“We suggest using the following sets of sampling parameters depending on the mode and task type”
“its multilingual speech generation ability generalizes well beyond the public benchmark languages”
“We suggest using Temperature=0.6, TopP=0.95, TopK=20, and MinP=0 for thinking mode”
“Reinforcement learning scaled across million-agent environments with progressively complex task distributions for robust real-world…”
“For multimodal workloads, use --mm-encoder-tp-mode data for data-parallel vision encoding and --mm-processor-cache-type shm for…”
“Include "Please reason step by step, and put your final answer within \boxed{}." in the prompt.”
“With `do_sample_frames=True`, you can customize the `fps` value to set your desired video sampling rate.”
“longest_edgeparameter in the video_preprocessor_config file to 469,762,048”
“Reduce --max-cudagraph-capture-size (default is 512).”
“OpenRouter routes your request to one of them based on the routing mode you pick — Balanced (price + speed), Nitro (fastest), or Exacto…”
“Hugging Face Transformers contains a lightweight server which can be used for quick testing and moderate load deployment.”
“For production workloads or high-throughput scenarios, dedicated serving engines such as SGLang, KTransformers or vLLM are strongly…”
“vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs.”
“It is recommended to set the longest_edge parameter in the video_preprocessor_config file to 469,762,048”
1 more mentioned or evaluated
“KTransformers is a flexible framework for experiencing cutting-edge LLM inference optimizations with CPU-GPU heterogeneous computing.”
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Qwen3.5-397B-A17B architecture card — LLM Architecture Gallery explainer sebastianraschka.com
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| qwen.ai/blog?id=qwen3.5 | official blog | Qwen | not fetched |
| huggingface.co/Qwen/Qwen3.5-397B-A17B | model card | Qwen | read |
| arxiv.org/abs/2604.15804 | technical report | Qwen Team | read |
| huggingface.co/Qwen/Qwen3.5-122B-A10B | model card | Qwen | read |
| huggingface.co/Qwen/Qwen3.5-35B-A3B | model card | Qwen | read |
| github.com/QwenLM/Qwen3.5 | code repo | Qwen | not fetched |
| recipes.vllm.ai/Qwen/Qwen3.5-397B-A17B | vendor docs | vLLM | read |
| openrouter.ai/qwen/qwen3.5-397b-a17b | vendor docs | OpenRouter | read |
| huggingface.co/blog/mlabonne/qwen35 | third party analysis | Hugging Face community (mlabonne) | read |
| deeplearning.ai/the-batch/qwen-announces-new-open-weights-flagship-upda… | news | The Batch (DeepLearning.AI) | read |
| lmsys.org/blog/2026-06-15-next-generation-speculative-decoding-dflash-v2 | vendor docs | LMSYS | not fetched |
| lmsys.org/blog/2026-09-16-nvfp4-kv-cache | vendor docs | LMSYS | not fetched |