Model techniques map
ModelsQwen3.6-35B-A3B

Alibaba (Qwen) · released 2026-04-15

Qwen3.6-35B-A3B

What Qwen3.6-35B-A3B’s own documents say it is built from — every method with the sentence that describes it.

OpenRouterHugging Face

Curator’s note35B total / 3B active sparse MoE aimed at agentic coding, successor to Qwen3.5-35B-A3B, Apache 2.0. The dense Qwen3.6-27B followed on 2026-04-21 and is listed as a source here. The hosted Plus/Max/Flash tiers are closed weights and excluded.

Against the consensus recipe

Its documents state 2 of the 15 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something the documents do not say, not something the model lacks.

  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 9 of the 19 architecture features checked — 6 of them are not stated in its documents.

Documents say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_qwen3_5_moe.py, configuration_qwen3_5_moe.py) and config.json at revision 995ad96eac, by anthropic/claude-sonnet-5 on 2026-09-25.

FeatureCodeIts documentsEvidence
Grouped-query attention code only in its codenot stated

Full-attention layers project 16 query heads down to 2 KV heads, repeated via repeat_kv for grouped-query attention.

text_config.num_attention_heads = 16text_config.num_key_value_heads = 2
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_qwen3_5_moe.py · L759
key_states = repeat_kv(key, module.num_key_value_groups)
    value_states = repeat_kv(value, module.num_key_value_groups)
modeling_qwen3_5_moe.py · L736–737
Multi-head latent attention not in its codenot stated

No kv_lora_rank/q_lora_rank low-rank KV latent projection exists; attention uses standard separate k_proj/v_proj heads.

self.k_proj = nn.Linear(
            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
        )
modeling_qwen3_5_moe.py · L766–768
Sliding-window attention not in its codenot stated

There is no sliding_window config key or masking mechanism for local windows in the text model; full_attention layers use the plain causal mask.

causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "linear_attention": create_recurrent_attention_mask(**mask_kwargs),
            }
modeling_qwen3_5_moe.py · L1393–1396
Interleaved sliding-window and global attention not in its codenot stated

The layer_types pattern interleaves full_attention with linear_attention (a recurrent mixer), not with sliding-window attention, so there is no sliding/global interleaving.

text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"…
Indexer-selected sparse attention (DeepSeek Sparse Attention) not in its codenot stated

No indexer scoring module or top-k token selection mechanism (index_topk, index_n_heads) appears in the code or config.

Linear-attention or state-space layers alongside full attention code only in its codenot stated

layer_types mixes linear_attention (Gated DeltaNet) layers with full_attention softmax layers in a fixed 4-layer interleave pattern.

text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"…text_config.full_attention_interval = 4
if self.block_type == "linear_attention":
            self.linear_attn = Qwen3_5MoeGatedDeltaNet(config, layer_idx)
        elif self.block_type == "full_attention":
            self.self_attn = Qwen3_5MoeAttention(config, layer_idx)
modeling_qwen3_5_moe.py · L952–955
Gated DeltaNet layers in its code core

Qwen3_5MoeGatedDeltaNet implements the gated delta rule with a decay gate (A_log/dt_bias) and beta-weighted delta update, used for all linear_attention layers.

text_config.linear_num_key_heads = 16text_config.linear_conv_kernel_dim = 4
g = -self.A_log.float().exp() * F.softplus(a.float() + self.dt_bias)
modeling_qwen3_5_moe.py · L621
last_recurrent_state = last_recurrent_state * chunk_decay[:, :, i] + key[:, :, i].transpose(-1, -2) @ v_new
modeling_qwen3_5_moe.py · L427
Mamba-2 layers not in its codenot stated

The linear-attention layers implement Gated DeltaNet, not Mamba-2 SSD blocks; no mamba_num_heads/ssm_state_size mechanism is present.

Learnable attention sink not in its codenot stated

No learned per-head sink logit is added to the attention softmax denominator anywhere in the attention forward code.

Gated attention output in its code core

In Qwen3_5MoeAttention the attention output is multiplied by a sigmoid gate computed from a separate chunk of the query projection before the output projection.

text_config.attn_output_gate = true
attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = attn_output * torch.sigmoid(gate)
modeling_qwen3_5_moe.py · L821–822
QK normalization code only in its codenot stated

Queries and keys are each normalized per head with Qwen3_5MoeRMSNorm (q_norm/k_norm) before RoPE and the dot product in full-attention layers.

self.q_norm = Qwen3_5MoeRMSNorm(self.head_dim, eps=config.rms_norm_eps)  # unlike olmo, only on the head dim!
        self.k_norm = Qwen3_5MoeRMSNorm(
            self.head_dim, eps=config.rms_norm_eps
        )
modeling_qwen3_5_moe.py · L775–778
Partial RoPE code only in its codenot stated

partial_rotary_factor of 0.25 restricts RoPE to a quarter of the head dimension, with the rest passed through unrotated in apply_rotary_pos_emb.

text_config.partial_rotary_factor = 0.25text_config.rope_parameters = {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "partial_rotary_fa…
rotary_dim = cos.shape[-1]
    q_rot, q_pass = q[..., :rotary_dim], q[..., rotary_dim:]
    k_rot, k_pass = k[..., :rotary_dim], k[..., rotary_dim:]
modeling_qwen3_5_moe.py · L700–702
YaRN RoPE scaling documents disagree not in its code optional

rope_parameters.rope_type is set to "default", not "yarn", so no YaRN scaling is applied.

text_config.rope_parameters = {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "partial_rotary_fa…
Layers without positional encoding (NoPE) not in its codenot stated

There is no no_rope_layers/nope_layer_interval mechanism; RoPE (partial) is applied uniformly to every full_attention layer via the shared rotary embedding.

cos, sin = position_embeddings
        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
modeling_qwen3_5_moe.py · L800–801
Mixture of experts in its code core

Qwen3_5MoeSparseMoeBlock routes each token to num_experts_per_tok=8 of num_experts=256 expert MLPs via a learned TopKRouter.

text_config.num_experts = 256text_config.num_experts_per_tok = 8
router_top_value, router_indices = torch.topk(router_probs, self.top_k, dim=-1)  # (seq_len, top_k)
modeling_qwen3_5_moe.py · L897
Shared expert code only in its codenot stated

Each MoE block also has a shared_expert MLP with nonzero shared_expert_intermediate_size, gated by a sigmoid and added to the routed expert output for every token.

text_config.shared_expert_intermediate_size = 512
self.shared_expert = Qwen3_5MoeMLP(config, intermediate_size=config.shared_expert_intermediate_size)
        self.shared_expert_gate = torch.nn.Linear(config.hidden_size, 1, bias=False)
modeling_qwen3_5_moe.py · L909–910
Auxiliary-loss-free load balancing (selection bias) not in its codenot stated

The router computes plain softmax top-k without any e_score_correction_bias-style selection bias; load balancing instead uses an auxiliary loss (router_aux_loss_coef).

router_probs = torch.nn.functional.softmax(router_logits, dtype=torch.float, dim=-1)
        router_top_value, router_indices = torch.topk(router_probs, self.top_k, dim=-1)  # (seq_len, top_k)
modeling_qwen3_5_moe.py · L896–897
Multi-token prediction layers code unclearnot stated

The config declares mtp_num_hidden_layers=1, but no MTP module class or weight declaration for it appears in the given modeling file, so the mechanism cannot be confirmed from these files.

text_config.mtp_num_hidden_layers = 1
SwiGLU feed-forward code only in its codenot stated

Both the dense MLP and the expert MLPs compute silu(gate_proj(x)) * up_proj(x) before down_proj, i.e. SwiGLU with hidden_act=silu.

text_config.hidden_act = "silu"
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
modeling_qwen3_5_moe.py · L840

model architecture 7

token mixer 2

linear attention & state space 1

gated delta network 1

Gated DeltaNet core

hybrid layer stacking 1

Gated DeltaNet and Gated Attention core

channel mixer 2

mixture of experts 2

Mixture of Experts core
Gated DeltaNet MoE core

positional encoding 2

YaRN optional
RoPE scaling optional

multimodal architecture 1

Early fusion multimodal training core

optimization 1

training parallelism 1

Tensor Parallelism used

data curation 1

data sourcing 1

Video sampling parameters optional

post-training 1

filed at the root 1

Scalable RL at Agent Scale core

inference & serving 24

decoding strategy 3

Multi-Token Prediction optional
Presence Penalty optional
Task-specific sampling parameters optional

reasoning control 5

Default thinking mode default
Thinking mode selection default
Interleaved thinking between tool calls default
Cross-turn persistent reasoning history used
Disabling reasoning via chat-template configuration optional

KV cache management 5

Chunked prefill used
Prefix caching used
Prompt caching used
Language-model-only serving mode optional
Mamba prefix caching in align mode evaluated

inference quantization 4

FP8 KV-cache quantization used
FP8 optional
NVFP4 quantization optional
NVFP4 ModelOpt re-quantization optional

inference scheduling 1

Asynchronous scheduling used

inference kernel 1

CUDA Graph capture size reduction used

context management 2

Preserved thinking history mode used
Preserve thinking optional

agentic scaffolding 3

Automatic tool choice used
Qwen3 XML tool-call parser used
MCP tool configuration optional

software implementation 4

filed at the root 1

fastsafetensors used

inference engine 1

Dedicated serving engines optional

kernel & quantization library 1

Marlin MoE backend used

infrastructure service 1

Automatic provider failover core

evaluation 1

filed at the root 1

Prompt-based output standardization used

unfiled 7

Gated Attention core
Asynchronous RL training infrastructure with near-100% multimodal efficiency core
Gated Delta Networks with sparse Mixture-of-Experts core
Qwen3 reasoning parser used
Adequate output length recommendation optional
OpenRouter routing modes (Balanced, Nitro, Exacto) optional
Output length recommendation optional

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated document list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

DocumentKindPublisherStatus
qwen.ai/blog?id=qwen3.6-35b-a3bofficial blogQwennot fetched
qwen.ai/blog?id=qwen3.6official blogQwennot fetched
huggingface.co/Qwen/Qwen3.6-35B-A3Bmodel cardQwenread
huggingface.co/Qwen/Qwen3.6-27Bmodel cardQwenread
github.com/QwenLM/Qwen3.6code repoQwenread
recipes.vllm.ai/Qwen/Qwen3.6-35B-A3Bvendor docsvLLMread
recipes.vllm.ai/Qwen/Qwen3.6-27Bvendor docsvLLMread
openrouter.ai/qwen/qwen3.6-35b-a3bvendor docsOpenRouterread