MiMo-V2.5
Xiaomimaintaining long-context performance via learnable attention sink bias
2 more from this model
maintaining long-context performance via learnable attention sink bias
learnable attention sink bias
specific method · filed under model architecture
A learnable sink logit is one extra softmax entry per head with a learned score and no value, so the attention weights of the real tokens can add up to less than one. It adds one number per head per layer and no bytes to the KV cache.
Written for the explainer Attention sinks and output gates.
Also called Attention sink bias.
Checked as “Learnable attention sink” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| DeepSeek-V4.1-Flash-1017 | in its code | not stated | verdict and linesAttention calls pass a learned attn_sink tensor into the sparse MLA kernels (sinks=self.attn_sink), implementing a learned per-head sink logit absorbing probability mass in the softmax denominator. | code → |
| MiMo-V2.6-Flash | in its code | not stated | add_swa_attention_sink_bias=true · add_full_attention_sink_bias=falseLanguage-model attention has a learned per-head attention_sink_bias parameter appended as an extra logit column in eager attention softmax, enabled for SWA layers by add_swa_attention_sink_bias=true. add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false | code → |
| MiMo-V2.6-Pro | in its code | not stated | add_swa_attention_sink_bias=true · add_full_attention_sink_bias=falseA learned per-head attention_sink_bias parameter is added as an extra logit column in the softmax denominator, enabled for both full and SWA layers via add_full_attention_sink_bias/add_swa_attention_sink_bias. add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false | code → |
| DeepSeek-V4.1-Flash | in its code | not stated | verdict and linesAttention layers pass a learned attn_sink into the sparse MLA kernels as a per-head sink logit absorbing softmax probability mass. | code → |
| Hy4-preview | in its code | not stated | learnable_sink=true · learnable_sink_init=0.0Each attention head has a learned sink parameter that is concatenated as an extra logit column before the softmax and then dropped from the output probabilities, enabled by learnable_sink. learnable_sink = truelearnable_sink_init = 0.0 | code → |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | verdict and linesEach attention layer has a per-head learnable sink parameter concatenated into the softmax logits before normalization, gpt-oss style. | code → |
| DeepSeek-V4-Pro-0813 | in its code | not stated | verdict and linesEach attention layer has a per-head learnable sink parameter concatenated as an extra logit column before the softmax, matching gpt-oss style sinks | code → |
| DeepSeek-V4-Flash-0731 | in its code | not stated | verdict and linesEach attention layer has a per-head learnable sink parameter concatenated into the softmax denominator before being dropped from the output. | code → |
| DeepSeek-V4-Flash | in its code | not stated | verdict and linesEach attention layer has a per-head learnable sink parameter concatenated into the softmax denominator before being dropped, exactly as in gpt-oss. | code → |
| DeepSeek-V4-Pro | in its code | not stated | verdict and linesEach attention layer has a per-head learnable sink parameter concatenated into the softmax denominator in eager_attention_forward. | code → |
| MiMo-V2.5 | in its code | adopted | add_swa_attention_sink_bias=true · add_full_attention_sink_bias=falseSWA layers get a learned per-head attention_sink_bias parameter concatenated as an extra logit column in the softmax denominator, enabled by add_swa_attention_sink_bias=true. add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false | code → |
| MiMo-V2.5-Pro | in its code | adopted | add_swa_attention_sink_bias=true · add_full_attention_sink_bias=falseSliding-window layers add a learned per-head attention_sink_bias as an extra logit column in the softmax denominator (enabled via add_swa_attention_sink_bias=true). add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false | code → |
| gpt-oss-120b | in its code | not stated | verdict and linesEach attention layer has a learned per-head sink parameter concatenated into the softmax denominator | code → |
| Mistral-Large-4 | not in its code | not stated | verdict and linesThere is no learned per-head sink logit added to the attention softmax denominator anywhere in the attention or eager_attention_forward code. | code → |
| LFM3-1.7B | not in its code | not stated | verdict and linesNo learned per-head sink logit is added to the attention softmax denominator anywhere in the code. | code → |
| MiniMax-M3.1-Flash | not in its code | not stated | verdict and linesThe eager attention forward computes a plain softmax over query-key scores with no extra learned sink logit column. | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | code unclear | not stated | verdict and linesNo attention-sink related keys or code are present in the given config/files to confirm or rule out a learned sink logit in the attention layers. | code → |
| Step-5-Preview | not in its code | not stated | verdict and linesNo learned per-head sink logit exists; sinks are explicitly rejected/unsupported in the DSA attention implementation. | code → |
| Kolibri-1 | not in its code | not stated | verdict and linesThe attention softmax has no extra learned sink logit column; eager_attention_forward only computes query-key-value softmax. | code → |
| GLM-5.3-Flash | not in its code | not stated | verdict and linesNo learned per-head sink logit or extra softmax denominator column exists anywhere in the attention implementations (MLA/DSA eager_attention_forward has no sink term). | code → |
| Qwen3.8-Flash-Next | not in its code | not stated | verdict and linesNo per-head learned sink logit is added to the attention softmax anywhere in the code. | code → |
| GLM-5.3 | not in its code | not stated | verdict and linesNo learned per-head sink logit is added to the attention softmax anywhere in the attention or eager_attention_forward code. | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | verdict and linesThe eager/sdpa attention implementation has no extra learned sink logit column added to the softmax denominator. | code → |
| Xiaomi-Robotics-1 | not in its code | not stated | verdict and linesThe DiT policy head has a learned 'sink' embedding concatenated as a token, not a per-head sink logit added to the attention softmax denominator; the language model's attention has no sink logit mechanism at all. | code → |
| Laguna-S-2.1 | not in its code | not stated | verdict and linesThe attention forward and eager_attention_forward implementation have no learned per-head sink logit added to the softmax denominator. | code → |
| Kimi K3 | not in its code | not stated | verdict and linesNo per-head learned sink logit is added to the attention softmax denominator anywhere in the eager or MLA attention implementations. | code → |
| Inkling | not in its code | not stated | verdict and linesNo learned per-head sink logit is added to the attention softmax denominator anywhere in the attention or kernel code. | code → |
| Hy3 | not in its code | not stated | verdict and linesThe attention forward computes a standard softmax over query-key scores with no extra learned sink logit column. | code → |
| GLM-5.2 | not in its code | not stated | verdict and linesNo learned per-head sink logit is added to the softmax denominator anywhere in the attention or indexer code. | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | not in its code | not stated | verdict and linesNo learned per-head sink logit is added to attention weights in eager_attention_forward or elsewhere; the softmax denominator is computed over only the key positions. | code → |
| MiniMax-M3 | not in its code | not stated | verdict and linesThe attention softmax has no extra learned sink logit; eager_attention_forward only uses the standard query/key/value matmuls. | code → |
| Step-3.7-Flash | not in its code | not stated | sink=falseThe config declares a 'sink' flag set to false and no per-head sink logit parameter or extra softmax column exists in the attention code. text_config.sink = false | code → |
| MolmoAct 2 | not in its code | not stated | verdict and linesNo learned per-head sink logit is added to the attention softmax denominator anywhere in the attention forward implementation. | code → |
| Qwen3.6-35B-A3B | not in its code | not stated | verdict and linesNo per-head learned sink logit is added to attention scores anywhere in the attention forward pass. | code → |
| Gemma 4 31B | not in its code | not stated | verdict and linesNo learned per-head sink logit is added to the attention softmax denominator anywhere in the attention implementation. | code → |
| Qwen3.5-397B-A17B | not in its code | not stated | verdict and linesNo learned per-head sink logit or extra softmax denominator column appears in the attention code. | code → |
| DeepSeek-V3.2 | not in its code | not stated | verdict and linesThe attention softmax implementation (eager_attention_forward) has no extra learned sink logit term added to the denominator. | code → |
| X-VLA | not in its code | not stated | verdict and linesNone of the attention modules (Florence2Attention, Florence2SdpaAttention, Florence2FlashAttention2, or the transformer.py Attention) add a learned per-head sink logit to the softmax denominator. | code → |
| OpenVLA / OpenVLA-OFT | not in its code | not stated | verdict and linesNo learned per-head sink logit mechanism is implemented anywhere in the provided modeling code. | code → |
4 quotes from 4 sources, one card per model, each opening with its strongest.
maintaining long-context performance via learnable attention sink bias
maintaining long-context performance via learnable attention sink bias
learnable attention sink bias
via learnable attention sink bias
Picked by hand, not extracted: where to read more, not evidence for anything on this page.