Open Models Almanac
Methods

specific method · not yet filed

Gated Attention

Gated attention multiplies a head's output by a factor computed from the current token, before the output projection. In the one study that tested variants, only gates specific to each head, reading the current token and using a plain sigmoid cleared the first-token sink.

Written for the explainer Attention sinks and output gates.

sources
3
models
2
lab adopt it
1
strongest
core

In practice

Checked as “Gated attention output” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
LFM3-1.7Bin its codenot stated
model_type=lfm3

use_attention_gate is true, so g_proj produces a per-head sigmoid gate that multiplies the attention output before the output projection.

model_type = "lfm3"
if self.g_proj is not None:
            attn_output = attn_output * torch.sigmoid(self.g_proj(hidden_states)).unsqueeze(-1)
modeling_lfm3.py · L280–281
Step-5-Previewin its codenot stated
use_head_wise_attn_gate=true

use_head_wise_attn_gate=true enables a per-head sigmoid gate (g_proj) multiplied into the attention output before the output projection.

text_config.use_head_wise_attn_gate = true
sigmoid = tl.div_rn(1.0, 1.0 + tl.extra.cuda.libdevice.exp(-gate))
attention_gate.py · L35
attn_output = headwise_sigmoid_gate(
                attn_output, extra_dims, slot_mapping=decode_slots
            )
model.py · L1413–1415
Hy4-previewin its codenot stated
gated_mla=true · gating_type=elementwise

An input-dependent gate projection produces per-head/per-element values passed through sigmoid and multiplied into the attention output before the output projection, consistent with gated_mla=true.

gated_mla = truegating_type = "elementwise"
gate_states = self.gate_proj(hidden_states).view(batch_size, seq_length, -1, self.gate_projection_size)
modeling_hy_v4.py · L408
attn_output = attn_output * torch.sigmoid(gate_states)
modeling_hy_v4.py · L478
Qwen3.8-Flash-Nextin its codenot stated
verdict and lines

Attention output is multiplied by a sigmoid gate derived from a chunk of the q_proj output before the output projection.

attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = attn_output * torch.sigmoid(gate)
modeling_qwen4_exp.py · L884–885
Laguna-S-2.1in its codenot stated
gating=per-head

Attention output is multiplied by a softplus (not sigmoid, but input-dependent) gate computed from g_proj before the output projection, per-head as configured by gating='per-head'.

gating = "per-head"
gate = F.softplus(self.g_proj(hidden_states).float()).to(attn_output.dtype)
        if self.gate_per_head:
            attn_output = (attn_output.view(*input_shape, -1, self.head_dim) * gate.unsqueeze(-1)).view(
                *input_shape, -1
            )
modeling_laguna.py · L400–404
Kimi K3in its codenot stated
mla_use_output_gate=true

In KimiMLAAttention, when mla_use_output_gate is true, a sigmoid gate computed from hidden_states (g_proj) multiplies the attention output before the final projection.

text_config.mla_use_output_gate = true
if self.use_output_gate:
            g = self.g_proj(hidden_states).sigmoid()
            attn_output = attn_output * g
        attn_output = self.o_proj(attn_output)
modeling_kimi_linear.py · L470–473
Step-3.7-Flashin its codenot stated
use_head_wise_attn_gate=true

The attention output is multiplied element-wise by a sigmoid of a learned gate projection (g_proj) before the output projection.

text_config.use_head_wise_attn_gate = true
attn_output = attn_output.view(*output_shape[:-1], self.num_heads, self.head_dim)
        attn_output = attn_output * gate_states.unsqueeze(-1).sigmoid()
        attn_output = attn_output.view(*output_shape)
modeling_step3p7.py · L740–742
Qwen3.6-35B-A3Bin its codeadopted
attn_output_gate=true

Attention output is multiplied by a sigmoid gate derived from an input-dependent projection before the output projection, enabled via attn_output_gate.

text_config.attn_output_gate = true
attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = attn_output * torch.sigmoid(gate)
modeling_qwen3_5_moe.py · L801–802
Qwen3.5-397B-A17Bin its codeadopted
attn_output_gate=true

Attention output is multiplied by a sigmoid gate derived from a chunked q_proj output before the output projection.

text_config.attn_output_gate = true
attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = attn_output * torch.sigmoid(gate)
modeling_qwen3_5_moe.py · L801–802
Mistral-Large-4not in its codenot stated
verdict and lines

The attention output is reshaped and passed directly to o_proj with no sigmoid gating multiplication.

attn_output = attn_output.reshape(batch_size, seq_length, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights, topk_indices
modeling_mistral_large4.py · L491–493
MiniMax-M3.1-Flashnot in its codenot stated
attention_output_gate=false

attention_output_gate is explicitly false in text_config and no sigmoid gate multiplication on attention output appears in the attention forward.

text_config.attention_output_gate = false
DeepSeek-V4.1-Flash-1017code unclearnot stated
verdict and lines

The attention output projection code (DeepseekV4Attention / o_proj kernels) is not included in the given files, so whether an input-dependent sigmoid gate multiplies the attention output before the output projection cannot be determined.

NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bcode unclearnot stated
verdict and lines

No attn_output_gate/use_output_gate keys appear and the nemotron_h attention implementation is not included, so an input-dependent sigmoid output gate cannot be confirmed.

Kolibri-1not in its codenot stated
verdict and lines

Attention output goes directly from attn_output reshape to o_proj with no sigmoid gating multiplication applied.

        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights
modeling_kolibri1.py · L204–206
MiMo-V2.6-Flashnot in its codenot stated
verdict and lines

The language-model attention output goes straight through o_proj with no sigmoid gate multiplying it; no attn_output_gate/use_output_gate config or code exists for the LM.

MiMo-V2.6-Pronot in its codenot stated
verdict and lines

The attention output is passed directly to o_proj with no sigmoid gating multiplication before the output projection.

DeepSeek-V4.1-Flashcode unclearnot stated
verdict and lines

The attention/o_proj implementation (dsv41_o_proj) is imported from files not included here, so whether the attention output is multiplied by an input-dependent sigmoid gate cannot be confirmed from the given code.

GLM-5.3-Flashnot in its codenot stated
verdict and lines

The softmax (MLA/DSA) attention output is passed straight to o_proj with no sigmoid gate; gating by sigmoid only occurs inside the linear-attention RMSNormGated module (o_norm), which is a normalization gate on the linear-attention branch, not a gated attention-output mechanism on the softmax attention path.

        attn_output = attn_output.reshape(batch_size, seq_length, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights, topk_indices if self.next_skip_topk else None
modeling_glm5_next.py · L1234–1236
DeepSeek-V4-Flash-Vision-Expnot in its codenot stated
verdict and lines

The attention output is not multiplied by an input-dependent sigmoid gate before the output projection; instead it undergoes a rope conjugate rotation and then the grouped output projection directly.

grouped = attn_output.reshape(*input_shape, self.config.o_groups, -1)
        grouped = self.o_a_proj(grouped).flatten(2)
        output = self.o_b_proj(grouped)
modeling_deepseek_v4.py · L881–883
GLM-5.3not in its codenot stated
verdict and lines

The attention output is passed straight to o_proj with no sigmoid gating of attn_output by an input-dependent gate.

attn_output = attn_output.reshape(batch_size, seq_length, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights, topk_indices
modeling_glm_moe_dsa.py · L448–450
DeepSeek-V4-Pro-0813not in its codenot stated
verdict and lines

There is no sigmoid gate multiplying the attention output before the output projection; the attention output instead goes through grouped linear projections and a RoPE-conjugate rotation, with no gating mechanism present

NVIDIA-Nemotron-3.5-Lightning-30B-A3Bnot in its codenot stated
verdict and lines

The attention output (NemotronHAttention) is passed directly to o_proj with no sigmoid gate multiplication; gating only appears in the Mamba mixer's silu gate, not in softmax attention.

attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights
modeling_nemotron_h.py · L918–920
DeepSeek-V4-Flash-0731not in its codenot stated
verdict and lines

The attention output is not multiplied by an input-dependent sigmoid gate; it goes through RoPE un-rotation and a grouped output projection with no gating mechanism.

grouped = attn_output.reshape(*input_shape, self.config.o_groups, -1)
        grouped = self.o_a_proj(grouped).flatten(2)
        output = self.o_b_proj(grouped)
modeling_deepseek_v4.py · L881–883
Xiaomi-Robotics-1not in its codenot stated
verdict and lines

Neither the Qwen3VL text attention nor the DiT attention multiplies the attention output by an input-dependent sigmoid gate before the output projection; AdaLN gates (gate_msa/gate_mlp) scale the residual branch output, not an attention-internal gate, and apply to the action DiT head, not the language model.

Inklingnot in its codenot stated
verdict and lines

The attention output is passed directly to the output projection (wo_ud) with no sigmoid gate multiplication applied to it.

flat = attn_output.view(num_tokens, -1)
        output, _ = self.wo_ud(flat)
        return output
attention.py · L269–271
Hy3not in its codenot stated
verdict and lines

The attention output is passed directly to o_proj with no sigmoid gating multiplication before the output projection.

attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)
modeling_hy_v3.py · L265–266
GLM-5.2not in its codenot stated
verdict and lines

The attention output is passed straight to o_proj with no input-dependent sigmoid gate multiplication.

attn_output = attn_output.reshape(batch_size, seq_length, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights, topk_indices
modeling_glm_moe_dsa.py · L448–450
NVIDIA-Nemotron-3-Ultra-550B-A55Bnot in its codenot stated
verdict and lines

NemotronHAttention's output goes directly through o_proj with no sigmoid gate multiplying the attention output; gating (silu) is only used inside the Mamba2 mixer's RMSNormGated, not attention.

attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights
modeling_nemotron_h.py · L918–920
MiniMax-M3not in its codenot stated
attention_output_gate=false

attention_output_gate is set false in text_config and no sigmoid gating of attention output is implemented before o_proj.

text_config.attention_output_gate = false
MolmoAct 2not in its codenot stated
verdict and lines

The attention output is passed directly to attn_out with no sigmoid gating multiplication applied to it in MolmoAct2Attention.forward.

DeepSeek-V4-Flashnot in its codenot stated
verdict and lines

The attention output is rotated back (conjugate RoPE) and passed through the grouped output projection, but there is no input-dependent sigmoid gate multiplying the attention output.

DeepSeek-V4-Pronot in its codenot stated
verdict and lines

The attention output is not multiplied by an input-dependent sigmoid gate; instead it undergoes a conjugate RoPE rotation and a grouped output projection, with no sigmoid gating mechanism present.

grouped = attn_output.reshape(*input_shape, self.config.o_groups, -1)
        grouped = self.o_a_proj(grouped).flatten(2)
        output = self.o_b_proj(grouped)
modeling_deepseek_v4.py · L881–883
MiMo-V2.5not in its codenot stated
verdict and lines

No sigmoid gating of attention output before o_proj is implemented; attn_output goes straight from attention interface to o_proj.

MiMo-V2.5-Pronot in its codenot stated
verdict and lines

No sigmoid gating multiplication is applied to the attention output before o_proj; attn_output goes directly into o_proj.

attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)
modeling_mimo_v2.py · L349–350
Gemma 4 31Bnot in its codenot stated
verdict and lines

Attention output is passed directly to o_proj with no sigmoid gating multiplication (attn_output_gate/use_output_gate not present).

attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights
modeling_gemma4.py · L966–968
DeepSeek-V3.2not in its codenot stated
verdict and lines

The attention output is reshaped and passed straight to o_proj with no sigmoid gating multiplication applied.

attn_output = attn_output.reshape(batch_size, seq_length, -1).contiguous()
        attn_output = self.o_proj(attn_output)
modeling_deepseek_v32.py · L475–476
X-VLAnot in its codenot stated
verdict and lines

No sigmoid gate multiplies the attention output before the output projection in any attention class; outputs go straight from attn@v to out_proj/proj.

gpt-oss-120bnot in its codenot stated
verdict and lines

No sigmoid gate is applied to the attention output before the output projection; attn_output is reshaped and passed directly to o_proj

attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)
modeling_gpt_oss.py · L325–326
OpenVLA / OpenVLA-OFTnot in its codenot stated
verdict and lines

No sigmoid gating of attention output is implemented in the provided code; the language model is instantiated from a stock Llama config with no gate parameters referenced.

Documented in

Evidence

3 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
core 3
↑