Learnable attention sink bias

A learnable bias used to provide an attention sink for maintaining long-context performance.

Also called Attention sink bias, learnable attention sink.

sources: 5
models: 3
lab adopt it: 1
strongest: used

How sources treat it

One count per evidence span, weakest treatment to strongest.

used 4not used 1

Documented in

MiMo-V2.5 MiniMax-M3 MiMo-V2.5-Pro

In the code

Checked as “Learnable attention sink” against each model’s own modeling code and config.json, independently of its documents. A model’s page shows the quoted lines.

Model	Code	Its documents	What the code does
GLM-5.3-Flash	not in its code	not stated	No learned per-head sink logit or extra denominator column appears anywhere in eager_attention_forward or the attention modules.
DeepSeek-V4.1-Flash	in its code	not stated	Each attention layer has a learned per-head attn_sink parameter added as an extra logit in the sparse-MLA attention softmax denominator (supports_sink is True for the backend). attention.py#L349
Hy4-preview	in its code	not stated	Each attention head has a learned sink logit parameter concatenated to the attention logits before the softmax denominator. modeling_hy_v4.py#L300
DeepSeek-V4-Flash-0731	in its code	not stated	Each attention layer has a learnable per-head sink parameter concatenated to the logits before softmax, gpt-oss style. modeling_deepseek_v4.py#L736
NVIDIA-Nemotron-3-Ultra-550B-A55B	not in its code	not stated	No learned per-head sink logit is added to the attention softmax denominator anywhere in the attention code.
MiMo-V2.6-Flash	in its code	not stated	Language-model attention has a learned per-head attention_sink_bias parameter appended as an extra logit column in eager attention softmax, enabled for SWA layers by add_swa_attention_sink_bias=true. modeling_mimo_v2.py#L89
DeepSeek-V4-Flash	in its code	not stated	Each attention head has a learned sinks parameter concatenated as an extra logit column before the softmax denominator, unconditionally in the eager attention path. modeling_deepseek_v4.py#L736
MiMo-V2.5	in its code	used	SWA layers get a learned per-head attention_sink_bias parameter concatenated as an extra logit column in the softmax denominator, enabled by add_swa_attention_sink_bias=true. modeling_mimo_v2.py#L89
GLM-5.3	not in its code	not stated	There is no learned per-head sink logit added to the attention softmax denominator anywhere in the eager or MLA attention code.
Hy3	not in its code	not stated	The softmax attention computation has no extra learned sink logit column added to the denominator modeling_hy_v3.py#L201
GLM-5.2	not in its code	not stated	The attention softmax in eager_attention_forward has no extra learned sink logit column added to the denominator. modeling_glm_moe_dsa.py#L285
MiniMax-M3	not in its code	not used	The eager attention forward computes a plain softmax over query-key scores with no extra learned sink logit column. modeling_minimax_m3_vl.py#L362
DeepSeek-V3.2	not in its code	not stated	The attention softmax implementation (eager_attention_forward) has no extra learned sink logit added to the denominator.
DeepSeek-V4-Flash-Vision-Exp	in its code	not stated	Each attention layer has a learnable per-head sink parameter concatenated into the softmax denominator before being dropped. modeling_deepseek_v4.py#L736
DeepSeek-V4-Pro	in its code	not stated	Each attention layer has a per-head learnable sink parameter concatenated into the softmax denominator modeling_deepseek_v4.py#L736
DeepSeek-V4-Pro-0813	in its code	not stated	Each attention layer has a per-head learnable sink parameter concatenated into the softmax denominator before dropping it, gpt-oss style. modeling_deepseek_v4.py#L736
Gemma 4 31B	not in its code	not stated	No learned per-head sink logit is added to the attention softmax denominator anywhere in eager_attention_forward or Gemma4TextAttention.
Inkling	not in its code	not stated	No learned per-head sink logit is added to the attention softmax denominator anywhere in the attention kernel; the only 'sink' concept here is the MoE shared-expert sink, unrelated to attention.
Kimi K3	not in its code	not stated	No per-head learned sink logit is added to the attention softmax denominator anywhere in the eager or MLA attention implementations.
Laguna-S-2.1	not in its code	not stated	The attention softmax and mask construction have no extra learned sink logit column. modeling_laguna.py#L327
MiMo-V2.5-Pro	in its code	used	Sliding-window layers add a learned per-head attention_sink_bias as an extra logit column in the softmax denominator (enabled via add_swa_attention_sink_bias=true). modeling_mimo_v2.py#L84
MiMo-V2.6-Pro	in its code	not stated	A learned per-head attention_sink_bias parameter is added as an extra logit column in the softmax denominator, enabled for both full and SWA layers via add_full_attention_sink_bias/add_swa_attention_sink_bias. modeling_mimo_v2.py#L89
NVIDIA-Nemotron-3.5-Lightning-30B-A3B	not in its code	not stated	No per-head learned sink logit is added to the attention softmax denominator anywhere in the attention implementation.
Qwen3.5-397B-A17B	not in its code	not stated	No learned per-head sink logit is added to the attention softmax anywhere in the attention forward code.
Qwen3.6-35B-A3B	not in its code	not stated	No learned per-head sink logit is added to the attention softmax denominator anywhere in the attention forward code.
Qwen3.8-Flash-Next	not in its code	not stated	No learned per-head sink logit is added to the attention softmax denominator anywhere in the code.
Step-3.7-Flash	not in its code	not stated	The config declares sink: false and no per-head sink logit parameter or addition to the softmax denominator appears in the attention code.
gpt-oss-120b	in its code	not stated	Each attention layer has a learned per-head sink logit concatenated into the softmax denominator. modeling_gpt_oss.py#L251