specific method · not yet filed
Gated Attention
- sources
- 3
- models
- 2
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
core 3
Documented in
In the code
Checked as “Gated attention output” against each model’s own modeling code and config.json, independently of its documents. A model’s page shows the quoted lines.
| Model | Code | Its documents | What the code does |
|---|---|---|---|
| GLM-5.3-Flash | not in its code | not stated | The softmax-attention (Glm5NextTextAttention/MLA) path has no sigmoid gate on its output before o_proj; the only sigmoid output gate found is in the linear-attention (KDA) path via RMSNormGated, which is a separate mixer, not gated softmax attention output. |
| DeepSeek-V4.1-Flash | not in its code | not stated | No sigmoid gate is applied to the attention output before the o_proj; the Engram module has a sigmoid gate but that is a separate n-gram injection mechanism, not attention-output gating. |
| Hy4-preview | in its code | not stated | The attention output is multiplied elementwise by a sigmoid gate computed from a separate gate_proj before the output projection. modeling_hy_v4.py#L490 |
| DeepSeek-V4-Flash-0731 | not in its code | not stated | The attention output is only rotated back and projected through the grouped output projection; there is no sigmoid gate multiplying the attention output before o_proj. |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | not in its code | not stated | The attention module's output goes directly to o_proj with no sigmoid gating multiplication; gating is only used in the Mamba mixer's RMSNormGated, not in softmax attention output. modeling_nemotron_h.py#L888 |
| MiMo-V2.6-Flash | not in its code | not stated | The language-model attention output goes straight through o_proj with no sigmoid gate multiplying it; no attn_output_gate/use_output_gate config or code exists for the LM. |
| DeepSeek-V4-Flash | not in its code | not stated | There is no sigmoid gate multiplying the attention output before the output projection; attn_output goes through RoPE-undo and grouped projection only, with no input-dependent gating. |
| MiMo-V2.5 | not in its code | not stated | No sigmoid gating of attention output before o_proj is implemented; attn_output goes straight from attention interface to o_proj. |
| GLM-5.3 | not in its code | not stated | The attention output is passed straight to o_proj with no sigmoid gate multiplication before the output projection. modeling_glm_moe_dsa.py#L460 |
| Hy3 | not in its code | not stated | The attention output is passed directly to o_proj with no sigmoid gating multiplication applied modeling_hy_v3.py#L276 |
| GLM-5.2 | not in its code | not stated | The attention output is reshaped and passed directly to o_proj with no sigmoid gate multiplication anywhere in the forward pass. modeling_glm_moe_dsa.py#L460 |
| MiniMax-M3 | not in its code | not stated | attention_output_gate is set false in the config and no sigmoid gating of the attention output before o_proj appears in the attention forward. |
| DeepSeek-V3.2 | not in its code | not stated | The attention output is passed straight through o_proj with no sigmoid gating multiplication applied to attn_output. modeling_deepseek_v32.py#L489 |
| DeepSeek-V4-Flash-Vision-Exp | not in its code | not stated | There is no input-dependent sigmoid gate multiplying the attention output before the output projection; the attention output only goes through a rope-undo step and grouped output projection. |
| DeepSeek-V4-Pro | not in its code | not stated | No sigmoid gate is applied to the attention output before the output projection; the attention output goes through rope-conjugate rotation and the grouped output projection directly |
| DeepSeek-V4-Pro-0813 | not in its code | not stated | There is no input-dependent sigmoid gate multiplying the attention output before the output projection; the attn_output only goes through a rope-undo and grouped output projection. |
| Gemma 4 31B | not in its code | not stated | The attention output goes straight to o_proj with no sigmoid gate multiplication applied to attn_output before the output projection. |
| Inkling | not in its code | not stated | The attention output goes directly from the FA4 kernel to the output projection wo_ud with no sigmoid gating multiplication applied to it. attention.py#L269 |
| Kimi K3 | in its code | not stated | In KimiMLAAttention, when mla_use_output_gate is true, a sigmoid gate computed from hidden_states (g_proj) multiplies the attention output before the final projection. modeling_kimi_linear.py#L470 |
| Laguna-S-2.1 | in its code | not stated | Attention output is multiplied by a softplus (not sigmoid, but input-dependent) gate computed from hidden_states via g_proj, either per-head or per-element based on config.gating. modeling_laguna.py#L411 |
| MiMo-V2.5-Pro | not in its code | not stated | No sigmoid gating multiplication is applied to the attention output before o_proj; attn_output goes directly into o_proj. modeling_mimo_v2.py#L349 |
| MiMo-V2.6-Pro | not in its code | not stated | The attention output is passed directly to o_proj with no sigmoid gating multiplication before the output projection. |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | NemotronHAttention's output goes straight from attn_output to o_proj with no sigmoid gate multiplication; gating only appears in the Mamba mixer's RMSNormGated, which is not attention output gating. modeling_nemotron_h.py#L888 |
| Qwen3.5-397B-A17B | in its code | core | Attention output is multiplied by a sigmoid gate derived from a split of the q_proj output before the output projection. modeling_qwen3_5_moe.py#L821 |
| Qwen3.6-35B-A3B | in its code | core | In Qwen3_5MoeAttention the attention output is multiplied by a sigmoid gate computed from a separate chunk of the query projection before the output projection. modeling_qwen3_5_moe.py#L821 |
| Qwen3.8-Flash-Next | in its code | not stated | Attention output is multiplied by a sigmoid gate derived from a chunk of the q_proj output before the output projection. modeling_qwen4_exp.py#L897 |
| Step-3.7-Flash | in its code | not stated | Step3p7Attention computes a gate projection (g_proj) and multiplies the attention output by its sigmoid before the output projection. modeling_step3p7.py#L751 |
| gpt-oss-120b | not in its code | not stated | The attention output is passed straight to o_proj with no sigmoid gating multiplication. |
Evidence
3 spans quoted from the sources, strongest treatment first.
Gated Attention:- Number of Attention Heads: 32 for Q and 2 for KV
coremodel architecturein Qwen3.5-397B-A17BQwen
Gated Attention:- Number of Attention Heads: 16 for Q and 2 for KV
coremodel architecturein Qwen3.5-35B-A3BQwen
Gated Attention: - Number of Attention Heads: 16 for Q and 2 for KV
coremodel architecturein Qwen3.6-35B-A3BQwen