Alibaba (Qwen)
Qwen3.6-35B-A3B
What Qwen3.6-35B-A3B’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 35B / 3B active
- Context
- 256K
- Attention
- linear hybridGated DeltaNet linear layers + gated full GQA layers
- Experts
- 256 routed + 1 shared, top-8
- Input
- text, image, video
- Licence
- Apache 2.0
- Released
- 2026-04-15
Against the consensus recipe
Its documents state 4 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictionnot in its documents
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modedefault
- Multi-Token Predictionoptional
- FP8optional
Checked in its code
Its code shows 9 of the 19 architecture features checked — 6 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_qwen3_5_moe.py, configuration_qwen3_5_moe.py) and config.json at revision 345d467b11, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention code only | in its code | not stated | Full-attention layers project 16 query heads down to 2 KV heads, repeated via repeat_kv (grouped-query attention). show evidencetext_config.num_attention_heads = 16text_config.num_key_value_heads = 2 |
| Multi-head latent attention | not in its code | not stated | No kv_lora_rank/q_lora_rank low-rank KV latent mechanism exists; attention uses standard separate k_proj/v_proj with GQA instead. |
| Sliding-window attention | not in its code | not stated | There is no sliding_window config key or windowed mask logic in the attention code; full_attention layers use the standard causal mask. |
| Interleaved sliding-window and global attention | not in its code | not stated | layer_types interleaves linear_attention and full_attention, not sliding-window and global attention; no sliding-window mechanism exists. show evidence |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | not in its code | not stated | No indexer scoring or top-k token selection mechanism (index_topk, index_n_heads) appears in config or code. |
| Linear-attention or state-space layers alongside full attention code only | in its code | not stated | layer_types alternates linear_attention (Gated DeltaNet) layers with full_attention softmax layers every 4th layer per… show evidencelayer_types alternates linear_attention (Gated DeltaNet) layers with full_attention softmax layers every 4th layer per full_attention_interval. text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"… |
| Gated DeltaNet layers | in its code | adopted | Qwen3_5MoeGatedDeltaNet implements the gated delta rule with decay gate g and beta update, used in all linear_attention layers. show evidencetext_config.linear_num_key_heads = 16text_config.linear_conv_kernel_dim = 4 |
| Mamba-2 layers | not in its code | not stated | The linear-attention layers implement the gated delta rule (DeltaNet-style), not Mamba-2 SSD blocks; no mamba_num_heads or ssm_state_size… show the whole verdictThe linear-attention layers implement the gated delta rule (DeltaNet-style), not Mamba-2 SSD blocks; no mamba_num_heads or ssm_state_size config is present. |
| Learnable attention sink | not in its code | not stated | No per-head learned sink logit is added to attention scores anywhere in the attention forward pass. |
| Gated attention output | in its code | adopted | Attention output is multiplied by a sigmoid gate derived from an input-dependent projection before the output projection, enabled via… show evidenceAttention output is multiplied by a sigmoid gate derived from an input-dependent projection before the output projection, enabled via attn_output_gate. text_config.attn_output_gate = true |
| QK normalization code only | in its code | not stated | Queries and keys are each passed through a per-head RMSNorm (q_norm/k_norm) before rotary embedding and attention. show evidence |
| Partial RoPE code only | in its code | not stated | partial_rotary_factor of 0.25 restricts RoPE to a quarter of the head dimension, with the rest passed through unrotated (q_pass/k_pass). show evidencetext_config.partial_rotary_factor = 0.25text_config.rope_parameters.partial_rotary_factor = 0.25 |
| YaRN RoPE scaling documents disagree | not in its code | adopted | rope_parameters.rope_type is "default", not yarn, so no YaRN scaling is applied. show evidencetext_config.rope_parameters.rope_type = "default" |
| Layers without positional encoding (NoPE) | not in its code | not stated | All full_attention layers apply rotary embeddings uniformly via apply_rotary_pos_emb; there is no per-layer flag disabling RoPE for some… show the whole verdictAll full_attention layers apply rotary embeddings uniformly via apply_rotary_pos_emb; there is no per-layer flag disabling RoPE for some softmax attention layers (linear-attention layers lacking RoPE don't count). |
| Mixture of experts | in its code | adopted | Qwen3_5MoeTopKRouter routes each token to num_experts_per_tok=8 of num_experts=256 expert MLPs via softmax top-k routing. show evidencetext_config.num_experts = 256text_config.num_experts_per_tok = 8 |
| Shared expert code only | in its code | not stated | Qwen3_5MoeSparseMoeBlock contains a shared_expert MLP with nonzero shared_expert_intermediate_size that every token passes through, gated… show evidenceQwen3_5MoeSparseMoeBlock contains a shared_expert MLP with nonzero shared_expert_intermediate_size that every token passes through, gated and added to the routed output. text_config.shared_expert_intermediate_size = 512 |
| Auxiliary-loss-free load balancing (selection bias) | not in its code | not stated | The router uses only a softmax over logits with no e_score_correction_bias or topk_method; load balancing is done via the auxiliary loss… show evidenceThe router uses only a softmax over logits with no e_score_correction_bias or topk_method; load balancing is done via the auxiliary loss function, not a bias-based selection mechanism. |
| Multi-token prediction layers | code unclear | not stated | The config declares mtp_num_hidden_layers=1 and the code ignores mtp.* weights on load, but no MTP module class or forward logic is… show evidenceThe config declares mtp_num_hidden_layers=1 and the code ignores mtp.* weights on load, but no MTP module class or forward logic is present in the given files to confirm how it is built. text_config.mtp_num_hidden_layers = 1text_config.mtp_use_dedicated_embeddings = false |
| SwiGLU feed-forward code only | in its code | not stated | Both the shared/dense MLP and the routed experts compute silu(gate_proj(x)) * up_proj(x), a SwiGLU feed-forward, per hidden_act=silu. show evidencetext_config.hidden_act = "silu" |
token mixer 3
linear attention & state space 2
gated delta network 2
“Gated DeltaNet: - Number of Linear Attention Heads: 32 for V and 16 for QK”
“It uses the same gated delta networks hybrid attention as its MoE siblings”
hybrid layer stacking 1
“It uses a hybrid sparse mixture-of-experts architecture combining Gated DeltaNet linear attention with standard gated attention layers”
channel mixer 2
mixture of experts 2
“Mixture Of Experts - Number of Experts: 256 - Number of Activated Experts: 8 Routed + 1 Shared”
“sharing the same gated-delta-networks MoE architecture but with 35B total parameters and 3B activated (256 experts, 8 routed + 1 shared)”
positional encoding 3
“For longer inputs, apply YaRN RoPE scaling via --hf-overrides and raise --max-model-len.”
“we recommend using RoPE scaling techniques to handle long texts effectively.”
“Pick factor to match your real workload — 2.0 covers ~524K, 4.0 covers ~1M — since YaRN at higher factors degrades short-context quality.”
multimodal architecture 1
“Early fusion training on trillions of multimodal tokens achieves cross-generational parity with Qwen3 and outperforms Qwen3-VL models…”
training parallelism 1
“--tensor-parallel-size 2”
data sourcing 1
“"mm_processor_kwargs": {"fps": 2, "do_sample_frames": True}”
filed at the root 1
“Reinforcement learning scaled across million-agent environments with progressively complex task distributions for robust real-world…”
decoding strategy 2
“MTP (multi-token prediction) is supported out of the box for low-latency decoding.”
“We suggest using the following sets of sampling parameters depending on the mode and task type”
reasoning control 4
“Qwen3.6 models operate in thinking mode by default, generating thinking content signified by thinking ... response before producing the…”
“By default, only the thinking blocks generated in handling the latest user message is retained, resulting in a pattern commonly as…”
“It includes integrated thinking mode with reasoning traces preserved across multi-turn conversations”
“Disable reasoning: add --default-chat-template-kwargs '{"enable_thinking": false}'.”
KV cache management 4
“--enable-prefix-caching”
“--enable-chunked-prefill”
“The chart below shows the average price customers are actually paying after prompt caching.”
“The following command skips the vision encoder and multimodal profiling to free up memory for additional KV cache: - vllm serve ...…”
inference quantization 4
“--kv-cache-dtype fp8”
“Qwen's official FP8 checkpoint”
“NVIDIA's ModelOpt NVFP4 checkpoint”
“NVIDIA's ModelOpt re-quantization: MLP linears drop to NVFP4 (W4A16) while the attention linears and KV cache stay FP8”
inference scheduling 1
“--async-scheduling”
inference kernel 1
“reduce --max-cudagraph-capture-size(default 512)”
context management 1
“A new feature retains thinking context across conversation history, streamlining iterative development and reducing overhead.”
agentic scaffolding 3
“--enable-auto-tool-choice”
“--tool-call-parser qwen3_xml”
“To define the available tools, you can use the MCP configuration file, use the integrated tool of Qwen-Agent, or integrate other tools by…”
1 more mentioned or evaluated
KV cache management “Prefix Caching (Mamba): currently experimental in "align" mode.”
filed at the root 1
“--load-format fastsafetensors”
inference engine 1
“For production workloads or high-throughput scenarios, dedicated serving engines such as SGLang, KTransformers or vLLM are strongly…”
kernel & quantization library 1
“--moe-backend marlin”
infrastructure service 1
“OpenRouter monitors every provider continuously and automatically retries on the next-best provider when one returns an error.”
filed at the root 1
“We recommend using prompts to standardize model outputs when benchmarking.”
other 1
filed at the root 1
“For supported frameworks, you can adjust the presence_penaltyparameter between 0 and 2 to reduce endless repetitions.”
unfiled 7
“Gated Attention: - Number of Attention Heads: 16 for Q and 2 for KV”
“Near-100% multimodal training efficiency compared to text-only training and asynchronous RL frameworks supporting massive-scale agent…”
“Gated Delta Networks combined with sparse Mixture-of-Experts deliver high-throughput inference with minimal latency and cost overhead.”
“--reasoning-parser qwen3”
“We recommend using an output length of 32,768 tokens for most queries.”
“OpenRouter routes your request to one of them based on the routing mode you pick — Balanced (price + speed), Nitro (fastest), or Exacto…”
“We recommend using an output length of 32,768 tokens for most queries.”
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Qwen3.6-35B-A3B architecture card — LLM Architecture Gallery explainer sebastianraschka.com
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| qwen.ai/blog?id=qwen3.6-35b-a3b | official blog | Qwen | not fetched |
| qwen.ai/blog?id=qwen3.6 | official blog | Qwen | not fetched |
| huggingface.co/Qwen/Qwen3.6-35B-A3B | model card | Qwen | read |
| huggingface.co/Qwen/Qwen3.6-27B | model card | Qwen | read |
| github.com/QwenLM/Qwen3.6 | code repo | Qwen | read |
| recipes.vllm.ai/Qwen/Qwen3.6-35B-A3B | vendor docs | vLLM | read |
| recipes.vllm.ai/Qwen/Qwen3.6-27B | vendor docs | vLLM | read |
| openrouter.ai/qwen/qwen3.6-35b-a3b | vendor docs | OpenRouter | read |