Open Models Almanac
ModelsQwen3.6-35B-A3B

Alibaba (Qwen)

Qwen3.6-35B-A3B

What Qwen3.6-35B-A3B’s own documents say it is built from — every method with the sentence that describes it.

Parameters
35B / 3B active
Context
256K
Attention
linear hybridGated DeltaNet linear layers + gated full GQA layers
Experts
256 routed + 1 shared, top-8
Input
text, image, video
Licence
Apache 2.0
Released
2026-04-15

Against the consensus recipe

Its documents state 4 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 9 of the 19 architecture features checked — 6 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_qwen3_5_moe.py, configuration_qwen3_5_moe.py) and config.json at revision 345d467b11, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention code only in its codenot stated

Full-attention layers project 16 query heads down to 2 KV heads, repeated via repeat_kv (grouped-query attention).

show evidencetext_config.num_attention_heads = 16text_config.num_key_value_heads = 2
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_qwen3_5_moe.py · L739
Multi-head latent attention not in its codenot stated

No kv_lora_rank/q_lora_rank low-rank KV latent mechanism exists; attention uses standard separate k_proj/v_proj with GQA instead.

Sliding-window attention not in its codenot stated

There is no sliding_window config key or windowed mask logic in the attention code; full_attention layers use the standard causal mask.

Interleaved sliding-window and global attention not in its codenot stated

layer_types interleaves linear_attention and full_attention, not sliding-window and global attention; no sliding-window mechanism exists.

show evidence
causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "linear_attention": create_recurrent_attention_mask(**mask_kwargs),
            }
modeling_qwen3_5_moe.py · L1346–1349
Indexer-selected sparse attention (DeepSeek Sparse Attention) not in its codenot stated

No indexer scoring or top-k token selection mechanism (index_topk, index_n_heads) appears in config or code.

Linear-attention or state-space layers alongside full attention code only in its codenot stated

layer_types alternates linear_attention (Gated DeltaNet) layers with full_attention softmax layers every 4th layer per…

show evidence

layer_types alternates linear_attention (Gated DeltaNet) layers with full_attention softmax layers every 4th layer per full_attention_interval.

text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"…
if self.block_type == "linear_attention":
            self.linear_attn = Qwen3_5MoeGatedDeltaNet(config, layer_idx)
        elif self.block_type == "full_attention":
            self.self_attn = Qwen3_5MoeAttention(config, layer_idx)
modeling_qwen3_5_moe.py · L932–935
Gated DeltaNet layers in its codeadopted

Qwen3_5MoeGatedDeltaNet implements the gated delta rule with decay gate g and beta update, used in all linear_attention layers.

show evidencetext_config.linear_num_key_heads = 16text_config.linear_conv_kernel_dim = 4
last_recurrent_state = last_recurrent_state * chunk_decay[:, :, i] + key[:, :, i].transpose(-1, -2) @ v_new
modeling_qwen3_5_moe.py · L406
g = -self.A_log.float().exp() * F.softplus(a.float() + self.dt_bias)
modeling_qwen3_5_moe.py · L602
Mamba-2 layers not in its codenot stated

The linear-attention layers implement the gated delta rule (DeltaNet-style), not Mamba-2 SSD blocks; no mamba_num_heads or ssm_state_size…

show the whole verdict

The linear-attention layers implement the gated delta rule (DeltaNet-style), not Mamba-2 SSD blocks; no mamba_num_heads or ssm_state_size config is present.

Learnable attention sink not in its codenot stated

No per-head learned sink logit is added to attention scores anywhere in the attention forward pass.

Gated attention output in its codeadopted

Attention output is multiplied by a sigmoid gate derived from an input-dependent projection before the output projection, enabled via…

show evidence

Attention output is multiplied by a sigmoid gate derived from an input-dependent projection before the output projection, enabled via attn_output_gate.

text_config.attn_output_gate = true
attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = attn_output * torch.sigmoid(gate)
modeling_qwen3_5_moe.py · L801–802
QK normalization code only in its codenot stated

Queries and keys are each passed through a per-head RMSNorm (q_norm/k_norm) before rotary embedding and attention.

show evidence
query_states = self.q_norm(query_states.view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_qwen3_5_moe.py · L776–777
Partial RoPE code only in its codenot stated

partial_rotary_factor of 0.25 restricts RoPE to a quarter of the head dimension, with the rest passed through unrotated (q_pass/k_pass).

show evidencetext_config.partial_rotary_factor = 0.25text_config.rope_parameters.partial_rotary_factor = 0.25
rotary_dim = cos.shape[-1]
    q_rot, q_pass = q[..., :rotary_dim], q[..., rotary_dim:]
    k_rot, k_pass = k[..., :rotary_dim], k[..., rotary_dim:]
modeling_qwen3_5_moe.py · L680–682
YaRN RoPE scaling documents disagree not in its codeadopted

rope_parameters.rope_type is "default", not yarn, so no YaRN scaling is applied.

show evidencetext_config.rope_parameters.rope_type = "default"
Layers without positional encoding (NoPE) not in its codenot stated

All full_attention layers apply rotary embeddings uniformly via apply_rotary_pos_emb; there is no per-layer flag disabling RoPE for some…

show the whole verdict

All full_attention layers apply rotary embeddings uniformly via apply_rotary_pos_emb; there is no per-layer flag disabling RoPE for some softmax attention layers (linear-attention layers lacking RoPE don't count).

Mixture of experts in its codeadopted

Qwen3_5MoeTopKRouter routes each token to num_experts_per_tok=8 of num_experts=256 expert MLPs via softmax top-k routing.

show evidencetext_config.num_experts = 256text_config.num_experts_per_tok = 8
router_top_value, router_indices = torch.topk(router_probs, self.top_k, dim=-1)  # (seq_len, top_k)
modeling_qwen3_5_moe.py · L877
Shared expert code only in its codenot stated

Qwen3_5MoeSparseMoeBlock contains a shared_expert MLP with nonzero shared_expert_intermediate_size that every token passes through, gated…

show evidence

Qwen3_5MoeSparseMoeBlock contains a shared_expert MLP with nonzero shared_expert_intermediate_size that every token passes through, gated and added to the routed output.

text_config.shared_expert_intermediate_size = 512
self.shared_expert = Qwen3_5MoeMLP(config, intermediate_size=config.shared_expert_intermediate_size)
        self.shared_expert_gate = torch.nn.Linear(config.hidden_size, 1, bias=False)
modeling_qwen3_5_moe.py · L889–890
Auxiliary-loss-free load balancing (selection bias) not in its codenot stated

The router uses only a softmax over logits with no e_score_correction_bias or topk_method; load balancing is done via the auxiliary loss…

show evidence

The router uses only a softmax over logits with no e_score_correction_bias or topk_method; load balancing is done via the auxiliary loss function, not a bias-based selection mechanism.

router_logits = F.linear(hidden_states, self.weight)  # (seq_len, num_experts)
        router_probs = torch.nn.functional.softmax(router_logits, dtype=torch.float, dim=-1)
modeling_qwen3_5_moe.py · L875–876
Multi-token prediction layers code unclearnot stated

The config declares mtp_num_hidden_layers=1 and the code ignores mtp.* weights on load, but no MTP module class or forward logic is…

show evidence

The config declares mtp_num_hidden_layers=1 and the code ignores mtp.* weights on load, but no MTP module class or forward logic is present in the given files to confirm how it is built.

text_config.mtp_num_hidden_layers = 1text_config.mtp_use_dedicated_embeddings = false
SwiGLU feed-forward code only in its codenot stated

Both the shared/dense MLP and the routed experts compute silu(gate_proj(x)) * up_proj(x), a SwiGLU feed-forward, per hidden_act=silu.

show evidencetext_config.hidden_act = "silu"
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
modeling_qwen3_5_moe.py · L820

token mixer 3

linear attention & state space 2

gated delta network 2

Gated DeltaNet core

“Gated DeltaNet: - Number of Linear Attention Heads: 32 for V and 16 for QK”

Gated DeltaNet hybrid attention core

“It uses the same gated delta networks hybrid attention as its MoE siblings”

hybrid layer stacking 1

Gated DeltaNet and full attention core

“It uses a hybrid sparse mixture-of-experts architecture combining Gated DeltaNet linear attention with standard gated attention layers”

channel mixer 2

mixture of experts 2

Mixture of experts core

“Mixture Of Experts - Number of Experts: 256 - Number of Activated Experts: 8 Routed + 1 Shared”

Gated DeltaNet MoE core

“sharing the same gated-delta-networks MoE architecture but with 35B total parameters and 3B activated (256 experts, 8 routed + 1 shared)”

positional encoding 3

YaRN optional

“For longer inputs, apply YaRN RoPE scaling via --hf-overrides and raise --max-model-len.”

RoPE scaling optional

“we recommend using RoPE scaling techniques to handle long texts effectively.”

YaRN scaling-factor selection optional

“Pick factor to match your real workload — 2.0 covers ~524K, 4.0 covers ~1M — since YaRN at higher factors degrades short-context quality.”

multimodal architecture 1

Early-fusion multimodal training core

“Early fusion training on trillions of multimodal tokens achieves cross-generational parity with Qwen3 and outperforms Qwen3-VL models…”

training parallelism 1

Tensor parallelism used

“--tensor-parallel-size 2”

data sourcing 1

Multimodal processor video-sampling parameters optional

“"mm_processor_kwargs": {"fps": 2, "do_sample_frames": True}”

filed at the root 1

Scalable RL at Agent Scale core

“Reinforcement learning scaled across million-agent environments with progressively complex task distributions for robust real-world…”

decoding strategy 2

Multi-Token Prediction optional

“MTP (multi-token prediction) is supported out of the box for low-latency decoding.”

Task-specific sampling-parameter presets optional

“We suggest using the following sets of sampling parameters depending on the mode and task type”

reasoning control 4

Thinking mode default

“Qwen3.6 models operate in thinking mode by default, generating thinking content signified by thinking ... response before producing the…”

Interleaved thinking default

“By default, only the thinking blocks generated in handling the latest user message is retained, resulting in a pattern commonly as…”

Persistent reasoning history used

“It includes integrated thinking mode with reasoning traces preserved across multi-turn conversations”

Disable reasoning via enable_thinking optional

“Disable reasoning: add --default-chat-template-kwargs '{"enable_thinking": false}'.”

KV cache management 4

Prefix caching used

“--enable-prefix-caching”

Chunked prefill used

“--enable-chunked-prefill”

Prompt caching used

“The chart below shows the average price customers are actually paying after prompt caching.”

Language-model-only serving mode optional

“The following command skips the vision encoder and multimodal profiling to free up memory for additional KV cache: - vllm serve ...…”

inference quantization 4

FP8 KV-cache quantization used

“--kv-cache-dtype fp8”

FP8 optional

“Qwen's official FP8 checkpoint”

NVFP4 quantization optional

“NVIDIA's ModelOpt NVFP4 checkpoint”

ModelOpt NVFP4 re-quantization optional

“NVIDIA's ModelOpt re-quantization: MLP linears drop to NVFP4 (W4A16) while the attention linears and KV cache stay FP8”

inference scheduling 1

Asynchronous inference and execution used

“--async-scheduling”

inference kernel 1

CUDA graph capture size reduction used

“reduce --max-cudagraph-capture-size(default 512)”

context management 1

Preserved thinking history used

“A new feature retains thinking context across conversation history, streamlining iterative development and reducing overhead.”

agentic scaffolding 3

Automatic tool choice used

“--enable-auto-tool-choice”

Qwen3 XML tool-call parser used

“--tool-call-parser qwen3_xml”

MCP configuration for tool definition optional

“To define the available tools, you can use the MCP configuration file, use the integrated tool of Qwen-Agent, or integrate other tools by…”

1 more mentioned or evaluated
Mamba prefix caching in align mode evaluated

KV cache management “Prefix Caching (Mamba): currently experimental in "align" mode.”

filed at the root 1

fastsafetensors used

“--load-format fastsafetensors”

inference engine 1

Dedicated serving engines optional

“For production workloads or high-throughput scenarios, dedicated serving engines such as SGLang, KTransformers or vLLM are strongly…”

kernel & quantization library 1

Marlin MoE backend used

“--moe-backend marlin”

infrastructure service 1

Automatic provider failover and retry core

“OpenRouter monitors every provider continuously and automatically retries on the next-best provider when one returns an error.”

filed at the root 1

Prompt-based output standardization used

“We recommend using prompts to standardize model outputs when benchmarking.”

filed at the root 1

Presence penalty adjustment optional

“For supported frameworks, you can adjust the presence_penaltyparameter between 0 and 2 to reduce endless repetitions.”

unfiled 7

Gated Attention core

“Gated Attention: - Number of Attention Heads: 16 for Q and 2 for KV”

Asynchronous RL training infrastructure with near-100% multimodal efficiency core

“Near-100% multimodal training efficiency compared to text-only training and asynchronous RL frameworks supporting massive-scale agent…”

Gated Delta Networks with sparse Mixture-of-Experts core

“Gated Delta Networks combined with sparse Mixture-of-Experts deliver high-throughput inference with minimal latency and cost overhead.”

Qwen3 reasoning parser used

“--reasoning-parser qwen3”

Adequate output length recommendation optional

“We recommend using an output length of 32,768 tokens for most queries.”

OpenRouter routing modes (Balanced, Nitro, Exacto) optional

“OpenRouter routes your request to one of them based on the routing mode you pick — Balanced (price + speed), Nitro (fastest), or Exacto…”

Output length recommendation optional

“We recommend using an output length of 32,768 tokens for most queries.”

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

SourceKindPublisherStatus
qwen.ai/blog?id=qwen3.6-35b-a3bofficial blogQwennot fetched
qwen.ai/blog?id=qwen3.6official blogQwennot fetched
huggingface.co/Qwen/Qwen3.6-35B-A3Bmodel cardQwenread
huggingface.co/Qwen/Qwen3.6-27Bmodel cardQwenread
github.com/QwenLM/Qwen3.6code repoQwenread
recipes.vllm.ai/Qwen/Qwen3.6-35B-A3Bvendor docsvLLMread
recipes.vllm.ai/Qwen/Qwen3.6-27Bvendor docsvLLMread
openrouter.ai/qwen/qwen3.6-35b-a3bvendor docsOpenRouterread
↑