Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attentionattention sink

methods · softmax attention

attention sink

4 methods filed at this node or below it, from the sources of 6 models.

In practice

Checked as “Learnable attention sink” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
DeepSeek-V4.1-Flash-1017in its codenot stated
verdict and lines

Attention calls pass a learned attn_sink tensor into the sparse MLA kernels (sinks=self.attn_sink), implementing a learned per-head sink logit absorbing probability mass in the softmax denominator.

out=decode_output,
                bmm1_scale=bmm1_scale,
                bmm2_scale=bmm2_scale,
                sinks=self.attn_sink,
                cum_seq_lens_q=decode_cu,
flashinfer_sparse.py · L540–544
MiMo-V2.6-Flashin its codenot stated
add_swa_attention_sink_bias=true · add_full_attention_sink_bias=false

Language-model attention has a learned per-head attention_sink_bias parameter appended as an extra logit column in eager attention softmax, enabled for SWA layers by add_swa_attention_sink_bias=true.

add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false
if sinks is not None:
        sinks = module.attention_sink_bias.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
        attn_weights = torch.cat([attn_weights, sinks], dim=-1)
modeling_mimo_v2.py · L89–91
self.attention_sink_bias = (
            nn.Parameter(torch.empty(self.num_attention_heads), requires_grad=False)
            if (
                (getattr(config, "add_full_attention_sink_bias", False) and not is_swa)
                or (getattr(config, "add_swa_attention_sink_bias", False) and is_swa)
            )
            else None
        )
modeling_mimo_v2.py · L268–275
MiMo-V2.6-Proin its codenot stated
add_swa_attention_sink_bias=true · add_full_attention_sink_bias=false

A learned per-head attention_sink_bias parameter is added as an extra logit column in the softmax denominator, enabled for both full and SWA layers via add_full_attention_sink_bias/add_swa_attention_sink_bias.

add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false
if sinks is not None:
        sinks = module.attention_sink_bias.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
        attn_weights = torch.cat([attn_weights, sinks], dim=-1)
modeling_mimo_v2.py · L89–91
DeepSeek-V4.1-Flashin its codenot stated
verdict and lines

Attention layers pass a learned attn_sink into the sparse MLA kernels as a per-head sink logit absorbing softmax probability mass.

                out=decode_output,
                bmm1_scale=bmm1_scale,
                bmm2_scale=bmm2_scale,
                sinks=self.attn_sink,
                cum_seq_lens_q=decode_cu,
                max_q_len=max_decode_query_len,
            )
flashinfer_sparse.py · L540–546
Hy4-previewin its codenot stated
learnable_sink=true · learnable_sink_init=0.0

Each attention head has a learned sink parameter that is concatenated as an extra logit column before the softmax and then dropped from the output probabilities, enabled by learnable_sink.

learnable_sink = truelearnable_sink_init = 0.0
sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_hy_v4.py · L289–290
self.sinks = nn.Parameter(torch.full((self.num_heads,), config.learnable_sink_init))
modeling_hy_v4.py · L373
DeepSeek-V4-Flash-Vision-Expin its codenot stated
verdict and lines

Each attention layer has a per-head learnable sink parameter concatenated into the softmax logits before normalization, gpt-oss style.

self.sinks = nn.Parameter(torch.empty(self.num_heads))
modeling_deepseek_v4.py · L807
sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_deepseek_v4.py · L744–745
DeepSeek-V4-Pro-0813in its codenot stated
verdict and lines

Each attention layer has a per-head learnable sink parameter concatenated as an extra logit column before the softmax, matching gpt-oss style sinks

self.sinks = nn.Parameter(torch.empty(self.num_heads))
modeling_deepseek_v4.py · L807
sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_deepseek_v4.py · L744–745
DeepSeek-V4-Flash-0731in its codenot stated
verdict and lines

Each attention layer has a per-head learnable sink parameter concatenated into the softmax denominator before being dropped from the output.

sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_deepseek_v4.py · L744–745
self.sinks = nn.Parameter(torch.empty(self.num_heads))
modeling_deepseek_v4.py · L807
DeepSeek-V4-Flashin its codenot stated
verdict and lines

Each attention layer has a per-head learnable sink parameter concatenated into the softmax denominator before being dropped, exactly as in gpt-oss.

sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_deepseek_v4.py · L744–745
self.sinks = nn.Parameter(torch.empty(self.num_heads))
modeling_deepseek_v4.py · L807
DeepSeek-V4-Proin its codenot stated
verdict and lines

Each attention layer has a per-head learnable sink parameter concatenated into the softmax denominator in eager_attention_forward.

sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_deepseek_v4.py · L744–745
self.sinks = nn.Parameter(torch.empty(self.num_heads))
modeling_deepseek_v4.py · L807
MiMo-V2.5in its codeadopted
add_swa_attention_sink_bias=true · add_full_attention_sink_bias=false

SWA layers get a learned per-head attention_sink_bias parameter concatenated as an extra logit column in the softmax denominator, enabled by add_swa_attention_sink_bias=true.

add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false
if sinks is not None:
        sinks = module.attention_sink_bias.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
        attn_weights = torch.cat([attn_weights, sinks], dim=-1)
modeling_mimo_v2.py · L89–91
self.attention_sink_bias = (
            nn.Parameter(torch.empty(self.num_attention_heads), requires_grad=False)
            if (
                (getattr(config, "add_full_attention_sink_bias", False) and not is_swa)
                or (getattr(config, "add_swa_attention_sink_bias", False) and is_swa)
            )
            else None
        )
modeling_mimo_v2.py · L268–275
MiMo-V2.5-Proin its codeadopted
add_swa_attention_sink_bias=true · add_full_attention_sink_bias=false

Sliding-window layers add a learned per-head attention_sink_bias as an extra logit column in the softmax denominator (enabled via add_swa_attention_sink_bias=true).

add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false
if sinks is not None:
        sinks = module.attention_sink_bias.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
        attn_weights = torch.cat([attn_weights, sinks], dim=-1)
modeling_mimo_v2.py · L84–86
self.attention_sink_bias = (
            nn.Parameter(torch.empty(self.num_attention_heads), requires_grad=False)
            if (
                (getattr(config, "add_full_attention_sink_bias", False) and not is_swa)
                or (getattr(config, "add_swa_attention_sink_bias", False) and is_swa)
            )
modeling_mimo_v2.py · L263–268
gpt-oss-120bin its codenot stated
verdict and lines

Each attention layer has a learned per-head sink parameter concatenated into the softmax denominator

self.sinks = nn.Parameter(torch.empty(config.num_attention_heads))
modeling_gpt_oss.py · L285
sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_gpt_oss.py · L243–244
Mistral-Large-4not in its codenot stated
verdict and lines

There is no learned per-head sink logit added to the attention softmax denominator anywhere in the attention or eager_attention_forward code.

LFM3-1.7Bnot in its codenot stated
verdict and lines

No learned per-head sink logit is added to the attention softmax denominator anywhere in the code.

MiniMax-M3.1-Flashnot in its codenot stated
verdict and lines

The eager attention forward computes a plain softmax over query-key scores with no extra learned sink logit column.

attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
modeling_minimax_m3_vl.py · L345
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bcode unclearnot stated
verdict and lines

No attention-sink related keys or code are present in the given config/files to confirm or rule out a learned sink logit in the attention layers.

Step-5-Previewnot in its codenot stated
verdict and lines

No learned per-head sink logit exists; sinks are explicitly rejected/unsupported in the DSA attention implementation.

if sinks is not None:
        raise NotImplementedError("Step5 DSA does not support attention sinks.")
sparse_attention.py · L182–183
Kolibri-1not in its codenot stated
verdict and lines

The attention softmax has no extra learned sink logit column; eager_attention_forward only computes query-key-value softmax.

    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
modeling_kolibri1.py · L128
GLM-5.3-Flashnot in its codenot stated
verdict and lines

No learned per-head sink logit or extra softmax denominator column exists anywhere in the attention implementations (MLA/DSA eager_attention_forward has no sink term).

Qwen3.8-Flash-Nextnot in its codenot stated
verdict and lines

No per-head learned sink logit is added to the attention softmax anywhere in the code.

GLM-5.3not in its codenot stated
verdict and lines

No learned per-head sink logit is added to the attention softmax anywhere in the attention or eager_attention_forward code.

NVIDIA-Nemotron-3.5-Lightning-30B-A3Bnot in its codenot stated
verdict and lines

The eager/sdpa attention implementation has no extra learned sink logit column added to the softmax denominator.

attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
    if attention_mask is not None:
        attn_weights = attn_weights + attention_mask

    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
modeling_nemotron_h.py · L856–860
Xiaomi-Robotics-1not in its codenot stated
verdict and lines

The DiT policy head has a learned 'sink' embedding concatenated as a token, not a per-head sink logit added to the attention softmax denominator; the language model's attention has no sink logit mechanism at all.

Laguna-S-2.1not in its codenot stated
verdict and lines

The attention forward and eager_attention_forward implementation have no learned per-head sink logit added to the softmax denominator.

Kimi K3not in its codenot stated
verdict and lines

No per-head learned sink logit is added to the attention softmax denominator anywhere in the eager or MLA attention implementations.

Inklingnot in its codenot stated
verdict and lines

No learned per-head sink logit is added to the attention softmax denominator anywhere in the attention or kernel code.

Hy3not in its codenot stated
verdict and lines

The attention forward computes a standard softmax over query-key scores with no extra learned sink logit column.

attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
modeling_hy_v3.py · L190
GLM-5.2not in its codenot stated
verdict and lines

No learned per-head sink logit is added to the softmax denominator anywhere in the attention or indexer code.

NVIDIA-Nemotron-3-Ultra-550B-A55Bnot in its codenot stated
verdict and lines

No learned per-head sink logit is added to attention weights in eager_attention_forward or elsewhere; the softmax denominator is computed over only the key positions.

attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
modeling_nemotron_h.py · L860
MiniMax-M3not in its codenot stated
verdict and lines

The attention softmax has no extra learned sink logit; eager_attention_forward only uses the standard query/key/value matmuls.

Step-3.7-Flashnot in its codenot stated
sink=false

The config declares a 'sink' flag set to false and no per-head sink logit parameter or extra softmax column exists in the attention code.

text_config.sink = false
MolmoAct 2not in its codenot stated
verdict and lines

No learned per-head sink logit is added to the attention softmax denominator anywhere in the attention forward implementation.

Qwen3.6-35B-A3Bnot in its codenot stated
verdict and lines

No per-head learned sink logit is added to attention scores anywhere in the attention forward pass.

Gemma 4 31Bnot in its codenot stated
verdict and lines

No learned per-head sink logit is added to the attention softmax denominator anywhere in the attention implementation.

Qwen3.5-397B-A17Bnot in its codenot stated
verdict and lines

No learned per-head sink logit or extra softmax denominator column appears in the attention code.

DeepSeek-V3.2not in its codenot stated
verdict and lines

The attention softmax implementation (eager_attention_forward) has no extra learned sink logit term added to the denominator.

attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
modeling_deepseek_v32.py · L273
X-VLAnot in its codenot stated
verdict and lines

None of the attention modules (Florence2Attention, Florence2SdpaAttention, Florence2FlashAttention2, or the transformer.py Attention) add a learned per-head sink logit to the softmax denominator.

OpenVLA / OpenVLA-OFTnot in its codenot stated
verdict and lines

No learned per-head sink logit mechanism is implemented anywhere in the provided modeling code.

In this branch 4

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 4

Learnable attention sink bias used · 4 sources · 4 quotes
Learnable attention sink token used · 2 sources · 2 quotes
2 single mentions
Attention sink used · 1 source · 1 quote
RoPE with attention sink unclear · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

↑