Qwen3.5-397B-A17B
Alibaba (Qwen)Gated Attention:- Number of Attention Heads: 32 for Q and 2 for KV
1 more from this model
Gated Attention:- Number of Attention Heads: 16 for Q and 2 for KV
specific method · not yet filed
Gated attention multiplies a head's output by a factor computed from the current token, before the output projection. In the one study that tested variants, only gates specific to each head, reading the current token and using a plain sigmoid cleared the first-token sink.
Written for the explainer Attention sinks and output gates.
Checked as “Gated attention output” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| LFM3-1.7B | in its code | not stated | model_type=lfm3use_attention_gate is true, so g_proj produces a per-head sigmoid gate that multiplies the attention output before the output projection. model_type = "lfm3" | code → |
| Step-5-Preview | in its code | not stated | use_head_wise_attn_gate=trueuse_head_wise_attn_gate=true enables a per-head sigmoid gate (g_proj) multiplied into the attention output before the output projection. text_config.use_head_wise_attn_gate = true | code → |
| Hy4-preview | in its code | not stated | gated_mla=true · gating_type=elementwiseAn input-dependent gate projection produces per-head/per-element values passed through sigmoid and multiplied into the attention output before the output projection, consistent with gated_mla=true. gated_mla = truegating_type = "elementwise" | code → |
| Qwen3.8-Flash-Next | in its code | not stated | verdict and linesAttention output is multiplied by a sigmoid gate derived from a chunk of the q_proj output before the output projection. | code → |
| Laguna-S-2.1 | in its code | not stated | gating=per-headAttention output is multiplied by a softplus (not sigmoid, but input-dependent) gate computed from g_proj before the output projection, per-head as configured by gating='per-head'. gating = "per-head" | code → |
| Kimi K3 | in its code | not stated | mla_use_output_gate=trueIn KimiMLAAttention, when mla_use_output_gate is true, a sigmoid gate computed from hidden_states (g_proj) multiplies the attention output before the final projection. text_config.mla_use_output_gate = true | code → |
| Step-3.7-Flash | in its code | not stated | use_head_wise_attn_gate=trueThe attention output is multiplied element-wise by a sigmoid of a learned gate projection (g_proj) before the output projection. text_config.use_head_wise_attn_gate = true | code → |
| Qwen3.6-35B-A3B | in its code | adopted | attn_output_gate=trueAttention output is multiplied by a sigmoid gate derived from an input-dependent projection before the output projection, enabled via attn_output_gate. text_config.attn_output_gate = true | code → |
| Qwen3.5-397B-A17B | in its code | adopted | attn_output_gate=trueAttention output is multiplied by a sigmoid gate derived from a chunked q_proj output before the output projection. text_config.attn_output_gate = true | code → |
| Mistral-Large-4 | not in its code | not stated | verdict and linesThe attention output is reshaped and passed directly to o_proj with no sigmoid gating multiplication. | code → |
| MiniMax-M3.1-Flash | not in its code | not stated | attention_output_gate=falseattention_output_gate is explicitly false in text_config and no sigmoid gate multiplication on attention output appears in the attention forward. text_config.attention_output_gate = false | code → |
| DeepSeek-V4.1-Flash-1017 | code unclear | not stated | verdict and linesThe attention output projection code (DeepseekV4Attention / o_proj kernels) is not included in the given files, so whether an input-dependent sigmoid gate multiplies the attention output before the output projection cannot be determined. | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | code unclear | not stated | verdict and linesNo attn_output_gate/use_output_gate keys appear and the nemotron_h attention implementation is not included, so an input-dependent sigmoid output gate cannot be confirmed. | code → |
| Kolibri-1 | not in its code | not stated | verdict and linesAttention output goes directly from attn_output reshape to o_proj with no sigmoid gating multiplication applied. | code → |
| MiMo-V2.6-Flash | not in its code | not stated | verdict and linesThe language-model attention output goes straight through o_proj with no sigmoid gate multiplying it; no attn_output_gate/use_output_gate config or code exists for the LM. | code → |
| MiMo-V2.6-Pro | not in its code | not stated | verdict and linesThe attention output is passed directly to o_proj with no sigmoid gating multiplication before the output projection. | code → |
| DeepSeek-V4.1-Flash | code unclear | not stated | verdict and linesThe attention/o_proj implementation (dsv41_o_proj) is imported from files not included here, so whether the attention output is multiplied by an input-dependent sigmoid gate cannot be confirmed from the given code. | code → |
| GLM-5.3-Flash | not in its code | not stated | verdict and linesThe softmax (MLA/DSA) attention output is passed straight to o_proj with no sigmoid gate; gating by sigmoid only occurs inside the linear-attention RMSNormGated module (o_norm), which is a normalization gate on the linear-attention branch, not a gated attention-output mechanism on the softmax attention path. | code → |
| DeepSeek-V4-Flash-Vision-Exp | not in its code | not stated | verdict and linesThe attention output is not multiplied by an input-dependent sigmoid gate before the output projection; instead it undergoes a rope conjugate rotation and then the grouped output projection directly. | code → |
| GLM-5.3 | not in its code | not stated | verdict and linesThe attention output is passed straight to o_proj with no sigmoid gating of attn_output by an input-dependent gate. | code → |
| DeepSeek-V4-Pro-0813 | not in its code | not stated | verdict and linesThere is no sigmoid gate multiplying the attention output before the output projection; the attention output instead goes through grouped linear projections and a RoPE-conjugate rotation, with no gating mechanism present | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | verdict and linesThe attention output (NemotronHAttention) is passed directly to o_proj with no sigmoid gate multiplication; gating only appears in the Mamba mixer's silu gate, not in softmax attention. | code → |
| DeepSeek-V4-Flash-0731 | not in its code | not stated | verdict and linesThe attention output is not multiplied by an input-dependent sigmoid gate; it goes through RoPE un-rotation and a grouped output projection with no gating mechanism. | code → |
| Xiaomi-Robotics-1 | not in its code | not stated | verdict and linesNeither the Qwen3VL text attention nor the DiT attention multiplies the attention output by an input-dependent sigmoid gate before the output projection; AdaLN gates (gate_msa/gate_mlp) scale the residual branch output, not an attention-internal gate, and apply to the action DiT head, not the language model. | code → |
| Inkling | not in its code | not stated | verdict and linesThe attention output is passed directly to the output projection (wo_ud) with no sigmoid gate multiplication applied to it. | code → |
| Hy3 | not in its code | not stated | verdict and linesThe attention output is passed directly to o_proj with no sigmoid gating multiplication before the output projection. | code → |
| GLM-5.2 | not in its code | not stated | verdict and linesThe attention output is passed straight to o_proj with no input-dependent sigmoid gate multiplication. | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | not in its code | not stated | verdict and linesNemotronHAttention's output goes directly through o_proj with no sigmoid gate multiplying the attention output; gating (silu) is only used inside the Mamba2 mixer's RMSNormGated, not attention. | code → |
| MiniMax-M3 | not in its code | not stated | attention_output_gate=falseattention_output_gate is set false in text_config and no sigmoid gating of attention output is implemented before o_proj. text_config.attention_output_gate = false | code → |
| MolmoAct 2 | not in its code | not stated | verdict and linesThe attention output is passed directly to attn_out with no sigmoid gating multiplication applied to it in MolmoAct2Attention.forward. | code → |
| DeepSeek-V4-Flash | not in its code | not stated | verdict and linesThe attention output is rotated back (conjugate RoPE) and passed through the grouped output projection, but there is no input-dependent sigmoid gate multiplying the attention output. | code → |
| DeepSeek-V4-Pro | not in its code | not stated | verdict and linesThe attention output is not multiplied by an input-dependent sigmoid gate; instead it undergoes a conjugate RoPE rotation and a grouped output projection, with no sigmoid gating mechanism present. | code → |
| MiMo-V2.5 | not in its code | not stated | verdict and linesNo sigmoid gating of attention output before o_proj is implemented; attn_output goes straight from attention interface to o_proj. | code → |
| MiMo-V2.5-Pro | not in its code | not stated | verdict and linesNo sigmoid gating multiplication is applied to the attention output before o_proj; attn_output goes directly into o_proj. | code → |
| Gemma 4 31B | not in its code | not stated | verdict and linesAttention output is passed directly to o_proj with no sigmoid gating multiplication (attn_output_gate/use_output_gate not present). | code → |
| DeepSeek-V3.2 | not in its code | not stated | verdict and linesThe attention output is reshaped and passed straight to o_proj with no sigmoid gating multiplication applied. | code → |
| X-VLA | not in its code | not stated | verdict and linesNo sigmoid gate multiplies the attention output before the output projection in any attention class; outputs go straight from attn@v to out_proj/proj. | code → |
| gpt-oss-120b | not in its code | not stated | verdict and linesNo sigmoid gate is applied to the attention output before the output projection; attn_output is reshaped and passed directly to o_proj | code → |
| OpenVLA / OpenVLA-OFT | not in its code | not stated | verdict and linesNo sigmoid gating of attention output is implemented in the provided code; the language model is instantiated from a stock Llama config with no gate parameters referenced. | code → |
3 quotes from 3 sources, one card per model, each opening with its strongest.
Gated Attention:- Number of Attention Heads: 32 for Q and 2 for KV
Gated Attention:- Number of Attention Heads: 16 for Q and 2 for KV
Gated Attention: - Number of Attention Heads: 16 for Q and 2 for KV