Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attentionattention sink

specific method · filed under model architecture

Learnable attention sink bias

A learnable sink logit is one extra softmax entry per head with a learned score and no value, so the attention weights of the real tokens can add up to less than one. It adds one number per head per layer and no bytes to the KV cache.

Written for the explainer Attention sinks and output gates.

Also called Attention sink bias.

sources
4
models
2
lab adopt it
1
strongest
used

In practice

Checked as “Learnable attention sink” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
DeepSeek-V4.1-Flash-1017in its codenot stated
verdict and lines

Attention calls pass a learned attn_sink tensor into the sparse MLA kernels (sinks=self.attn_sink), implementing a learned per-head sink logit absorbing probability mass in the softmax denominator.

out=decode_output,
                bmm1_scale=bmm1_scale,
                bmm2_scale=bmm2_scale,
                sinks=self.attn_sink,
                cum_seq_lens_q=decode_cu,
flashinfer_sparse.py · L540–544
MiMo-V2.6-Flashin its codenot stated
add_swa_attention_sink_bias=true · add_full_attention_sink_bias=false

Language-model attention has a learned per-head attention_sink_bias parameter appended as an extra logit column in eager attention softmax, enabled for SWA layers by add_swa_attention_sink_bias=true.

add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false
if sinks is not None:
        sinks = module.attention_sink_bias.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
        attn_weights = torch.cat([attn_weights, sinks], dim=-1)
modeling_mimo_v2.py · L89–91
self.attention_sink_bias = (
            nn.Parameter(torch.empty(self.num_attention_heads), requires_grad=False)
            if (
                (getattr(config, "add_full_attention_sink_bias", False) and not is_swa)
                or (getattr(config, "add_swa_attention_sink_bias", False) and is_swa)
            )
            else None
        )
modeling_mimo_v2.py · L268–275
MiMo-V2.6-Proin its codenot stated
add_swa_attention_sink_bias=true · add_full_attention_sink_bias=false

A learned per-head attention_sink_bias parameter is added as an extra logit column in the softmax denominator, enabled for both full and SWA layers via add_full_attention_sink_bias/add_swa_attention_sink_bias.

add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false
if sinks is not None:
        sinks = module.attention_sink_bias.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
        attn_weights = torch.cat([attn_weights, sinks], dim=-1)
modeling_mimo_v2.py · L89–91
DeepSeek-V4.1-Flashin its codenot stated
verdict and lines

Attention layers pass a learned attn_sink into the sparse MLA kernels as a per-head sink logit absorbing softmax probability mass.

                out=decode_output,
                bmm1_scale=bmm1_scale,
                bmm2_scale=bmm2_scale,
                sinks=self.attn_sink,
                cum_seq_lens_q=decode_cu,
                max_q_len=max_decode_query_len,
            )
flashinfer_sparse.py · L540–546
Hy4-previewin its codenot stated
learnable_sink=true · learnable_sink_init=0.0

Each attention head has a learned sink parameter that is concatenated as an extra logit column before the softmax and then dropped from the output probabilities, enabled by learnable_sink.

learnable_sink = truelearnable_sink_init = 0.0
sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_hy_v4.py · L289–290
self.sinks = nn.Parameter(torch.full((self.num_heads,), config.learnable_sink_init))
modeling_hy_v4.py · L373
DeepSeek-V4-Flash-Vision-Expin its codenot stated
verdict and lines

Each attention layer has a per-head learnable sink parameter concatenated into the softmax logits before normalization, gpt-oss style.

self.sinks = nn.Parameter(torch.empty(self.num_heads))
modeling_deepseek_v4.py · L807
sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_deepseek_v4.py · L744–745
DeepSeek-V4-Pro-0813in its codenot stated
verdict and lines

Each attention layer has a per-head learnable sink parameter concatenated as an extra logit column before the softmax, matching gpt-oss style sinks

self.sinks = nn.Parameter(torch.empty(self.num_heads))
modeling_deepseek_v4.py · L807
sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_deepseek_v4.py · L744–745
DeepSeek-V4-Flash-0731in its codenot stated
verdict and lines

Each attention layer has a per-head learnable sink parameter concatenated into the softmax denominator before being dropped from the output.

sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_deepseek_v4.py · L744–745
self.sinks = nn.Parameter(torch.empty(self.num_heads))
modeling_deepseek_v4.py · L807
DeepSeek-V4-Flashin its codenot stated
verdict and lines

Each attention layer has a per-head learnable sink parameter concatenated into the softmax denominator before being dropped, exactly as in gpt-oss.

sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_deepseek_v4.py · L744–745
self.sinks = nn.Parameter(torch.empty(self.num_heads))
modeling_deepseek_v4.py · L807
DeepSeek-V4-Proin its codenot stated
verdict and lines

Each attention layer has a per-head learnable sink parameter concatenated into the softmax denominator in eager_attention_forward.

sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_deepseek_v4.py · L744–745
self.sinks = nn.Parameter(torch.empty(self.num_heads))
modeling_deepseek_v4.py · L807
MiMo-V2.5in its codeadopted
add_swa_attention_sink_bias=true · add_full_attention_sink_bias=false

SWA layers get a learned per-head attention_sink_bias parameter concatenated as an extra logit column in the softmax denominator, enabled by add_swa_attention_sink_bias=true.

add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false
if sinks is not None:
        sinks = module.attention_sink_bias.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
        attn_weights = torch.cat([attn_weights, sinks], dim=-1)
modeling_mimo_v2.py · L89–91
self.attention_sink_bias = (
            nn.Parameter(torch.empty(self.num_attention_heads), requires_grad=False)
            if (
                (getattr(config, "add_full_attention_sink_bias", False) and not is_swa)
                or (getattr(config, "add_swa_attention_sink_bias", False) and is_swa)
            )
            else None
        )
modeling_mimo_v2.py · L268–275
MiMo-V2.5-Proin its codeadopted
add_swa_attention_sink_bias=true · add_full_attention_sink_bias=false

Sliding-window layers add a learned per-head attention_sink_bias as an extra logit column in the softmax denominator (enabled via add_swa_attention_sink_bias=true).

add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false
if sinks is not None:
        sinks = module.attention_sink_bias.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
        attn_weights = torch.cat([attn_weights, sinks], dim=-1)
modeling_mimo_v2.py · L84–86
self.attention_sink_bias = (
            nn.Parameter(torch.empty(self.num_attention_heads), requires_grad=False)
            if (
                (getattr(config, "add_full_attention_sink_bias", False) and not is_swa)
                or (getattr(config, "add_swa_attention_sink_bias", False) and is_swa)
            )
modeling_mimo_v2.py · L263–268
gpt-oss-120bin its codenot stated
verdict and lines

Each attention layer has a learned per-head sink parameter concatenated into the softmax denominator

self.sinks = nn.Parameter(torch.empty(config.num_attention_heads))
modeling_gpt_oss.py · L285
sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_gpt_oss.py · L243–244
Mistral-Large-4not in its codenot stated
verdict and lines

There is no learned per-head sink logit added to the attention softmax denominator anywhere in the attention or eager_attention_forward code.

LFM3-1.7Bnot in its codenot stated
verdict and lines

No learned per-head sink logit is added to the attention softmax denominator anywhere in the code.

MiniMax-M3.1-Flashnot in its codenot stated
verdict and lines

The eager attention forward computes a plain softmax over query-key scores with no extra learned sink logit column.

attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
modeling_minimax_m3_vl.py · L345
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bcode unclearnot stated
verdict and lines

No attention-sink related keys or code are present in the given config/files to confirm or rule out a learned sink logit in the attention layers.

Step-5-Previewnot in its codenot stated
verdict and lines

No learned per-head sink logit exists; sinks are explicitly rejected/unsupported in the DSA attention implementation.

if sinks is not None:
        raise NotImplementedError("Step5 DSA does not support attention sinks.")
sparse_attention.py · L182–183
Kolibri-1not in its codenot stated
verdict and lines

The attention softmax has no extra learned sink logit column; eager_attention_forward only computes query-key-value softmax.

    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
modeling_kolibri1.py · L128
GLM-5.3-Flashnot in its codenot stated
verdict and lines

No learned per-head sink logit or extra softmax denominator column exists anywhere in the attention implementations (MLA/DSA eager_attention_forward has no sink term).

Qwen3.8-Flash-Nextnot in its codenot stated
verdict and lines

No per-head learned sink logit is added to the attention softmax anywhere in the code.

GLM-5.3not in its codenot stated
verdict and lines

No learned per-head sink logit is added to the attention softmax anywhere in the attention or eager_attention_forward code.

NVIDIA-Nemotron-3.5-Lightning-30B-A3Bnot in its codenot stated
verdict and lines

The eager/sdpa attention implementation has no extra learned sink logit column added to the softmax denominator.

attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
    if attention_mask is not None:
        attn_weights = attn_weights + attention_mask

    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
modeling_nemotron_h.py · L856–860
Xiaomi-Robotics-1not in its codenot stated
verdict and lines

The DiT policy head has a learned 'sink' embedding concatenated as a token, not a per-head sink logit added to the attention softmax denominator; the language model's attention has no sink logit mechanism at all.

Laguna-S-2.1not in its codenot stated
verdict and lines

The attention forward and eager_attention_forward implementation have no learned per-head sink logit added to the softmax denominator.

Kimi K3not in its codenot stated
verdict and lines

No per-head learned sink logit is added to the attention softmax denominator anywhere in the eager or MLA attention implementations.

Inklingnot in its codenot stated
verdict and lines

No learned per-head sink logit is added to the attention softmax denominator anywhere in the attention or kernel code.

Hy3not in its codenot stated
verdict and lines

The attention forward computes a standard softmax over query-key scores with no extra learned sink logit column.

attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
modeling_hy_v3.py · L190
GLM-5.2not in its codenot stated
verdict and lines

No learned per-head sink logit is added to the softmax denominator anywhere in the attention or indexer code.

NVIDIA-Nemotron-3-Ultra-550B-A55Bnot in its codenot stated
verdict and lines

No learned per-head sink logit is added to attention weights in eager_attention_forward or elsewhere; the softmax denominator is computed over only the key positions.

attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
modeling_nemotron_h.py · L860
MiniMax-M3not in its codenot stated
verdict and lines

The attention softmax has no extra learned sink logit; eager_attention_forward only uses the standard query/key/value matmuls.

Step-3.7-Flashnot in its codenot stated
sink=false

The config declares a 'sink' flag set to false and no per-head sink logit parameter or extra softmax column exists in the attention code.

text_config.sink = false
MolmoAct 2not in its codenot stated
verdict and lines

No learned per-head sink logit is added to the attention softmax denominator anywhere in the attention forward implementation.

Qwen3.6-35B-A3Bnot in its codenot stated
verdict and lines

No per-head learned sink logit is added to attention scores anywhere in the attention forward pass.

Gemma 4 31Bnot in its codenot stated
verdict and lines

No learned per-head sink logit is added to the attention softmax denominator anywhere in the attention implementation.

Qwen3.5-397B-A17Bnot in its codenot stated
verdict and lines

No learned per-head sink logit or extra softmax denominator column appears in the attention code.

DeepSeek-V3.2not in its codenot stated
verdict and lines

The attention softmax implementation (eager_attention_forward) has no extra learned sink logit term added to the denominator.

attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
modeling_deepseek_v32.py · L273
X-VLAnot in its codenot stated
verdict and lines

None of the attention modules (Florence2Attention, Florence2SdpaAttention, Florence2FlashAttention2, or the transformer.py Attention) add a learned per-head sink logit to the softmax denominator.

OpenVLA / OpenVLA-OFTnot in its codenot stated
verdict and lines

No learned per-head sink logit mechanism is implemented anywhere in the provided modeling code.

Documented in

Evidence

4 quotes from 4 sources, one card per model, each opening with its strongest.

How sources treat it
used 4

MiMo-V2.5

Xiaomiadopted3 sources

maintaining long-context performance via learnable attention sink bias

usedmodel architecturein MiMo-V2.5Xiaomi
2 more from this model

maintaining long-context performance via learnable attention sink bias

usedmodel architecturein MiMo-V2.5Xiaomi

learnable attention sink bias

usedmodel architecturein MiMo-V2-FlashXiaomi

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑