methods · softmax attention
attention sink
4 methods filed at this node or below it, from the sources of 6 models.
In practice
Checked as “Learnable attention sink” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| DeepSeek-V4.1-Flash-1017 | in its code | not stated | verdict and linesAttention calls pass a learned attn_sink tensor into the sparse MLA kernels (sinks=self.attn_sink), implementing a learned per-head sink logit absorbing probability mass in the softmax denominator. | code → |
| MiMo-V2.6-Flash | in its code | not stated | add_swa_attention_sink_bias=true · add_full_attention_sink_bias=falseLanguage-model attention has a learned per-head attention_sink_bias parameter appended as an extra logit column in eager attention softmax, enabled for SWA layers by add_swa_attention_sink_bias=true. add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false | code → |
| MiMo-V2.6-Pro | in its code | not stated | add_swa_attention_sink_bias=true · add_full_attention_sink_bias=falseA learned per-head attention_sink_bias parameter is added as an extra logit column in the softmax denominator, enabled for both full and SWA layers via add_full_attention_sink_bias/add_swa_attention_sink_bias. add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false | code → |
| DeepSeek-V4.1-Flash | in its code | not stated | verdict and linesAttention layers pass a learned attn_sink into the sparse MLA kernels as a per-head sink logit absorbing softmax probability mass. | code → |
| Hy4-preview | in its code | not stated | learnable_sink=true · learnable_sink_init=0.0Each attention head has a learned sink parameter that is concatenated as an extra logit column before the softmax and then dropped from the output probabilities, enabled by learnable_sink. learnable_sink = truelearnable_sink_init = 0.0 | code → |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | verdict and linesEach attention layer has a per-head learnable sink parameter concatenated into the softmax logits before normalization, gpt-oss style. | code → |
| DeepSeek-V4-Pro-0813 | in its code | not stated | verdict and linesEach attention layer has a per-head learnable sink parameter concatenated as an extra logit column before the softmax, matching gpt-oss style sinks | code → |
| DeepSeek-V4-Flash-0731 | in its code | not stated | verdict and linesEach attention layer has a per-head learnable sink parameter concatenated into the softmax denominator before being dropped from the output. | code → |
| DeepSeek-V4-Flash | in its code | not stated | verdict and linesEach attention layer has a per-head learnable sink parameter concatenated into the softmax denominator before being dropped, exactly as in gpt-oss. | code → |
| DeepSeek-V4-Pro | in its code | not stated | verdict and linesEach attention layer has a per-head learnable sink parameter concatenated into the softmax denominator in eager_attention_forward. | code → |
| MiMo-V2.5 | in its code | adopted | add_swa_attention_sink_bias=true · add_full_attention_sink_bias=falseSWA layers get a learned per-head attention_sink_bias parameter concatenated as an extra logit column in the softmax denominator, enabled by add_swa_attention_sink_bias=true. add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false | code → |
| MiMo-V2.5-Pro | in its code | adopted | add_swa_attention_sink_bias=true · add_full_attention_sink_bias=falseSliding-window layers add a learned per-head attention_sink_bias as an extra logit column in the softmax denominator (enabled via add_swa_attention_sink_bias=true). add_swa_attention_sink_bias = trueadd_full_attention_sink_bias = false | code → |
| gpt-oss-120b | in its code | not stated | verdict and linesEach attention layer has a learned per-head sink parameter concatenated into the softmax denominator | code → |
| Mistral-Large-4 | not in its code | not stated | verdict and linesThere is no learned per-head sink logit added to the attention softmax denominator anywhere in the attention or eager_attention_forward code. | code → |
| LFM3-1.7B | not in its code | not stated | verdict and linesNo learned per-head sink logit is added to the attention softmax denominator anywhere in the code. | code → |
| MiniMax-M3.1-Flash | not in its code | not stated | verdict and linesThe eager attention forward computes a plain softmax over query-key scores with no extra learned sink logit column. | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | code unclear | not stated | verdict and linesNo attention-sink related keys or code are present in the given config/files to confirm or rule out a learned sink logit in the attention layers. | code → |
| Step-5-Preview | not in its code | not stated | verdict and linesNo learned per-head sink logit exists; sinks are explicitly rejected/unsupported in the DSA attention implementation. | code → |
| Kolibri-1 | not in its code | not stated | verdict and linesThe attention softmax has no extra learned sink logit column; eager_attention_forward only computes query-key-value softmax. | code → |
| GLM-5.3-Flash | not in its code | not stated | verdict and linesNo learned per-head sink logit or extra softmax denominator column exists anywhere in the attention implementations (MLA/DSA eager_attention_forward has no sink term). | code → |
| Qwen3.8-Flash-Next | not in its code | not stated | verdict and linesNo per-head learned sink logit is added to the attention softmax anywhere in the code. | code → |
| GLM-5.3 | not in its code | not stated | verdict and linesNo learned per-head sink logit is added to the attention softmax anywhere in the attention or eager_attention_forward code. | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | verdict and linesThe eager/sdpa attention implementation has no extra learned sink logit column added to the softmax denominator. | code → |
| Xiaomi-Robotics-1 | not in its code | not stated | verdict and linesThe DiT policy head has a learned 'sink' embedding concatenated as a token, not a per-head sink logit added to the attention softmax denominator; the language model's attention has no sink logit mechanism at all. | code → |
| Laguna-S-2.1 | not in its code | not stated | verdict and linesThe attention forward and eager_attention_forward implementation have no learned per-head sink logit added to the softmax denominator. | code → |
| Kimi K3 | not in its code | not stated | verdict and linesNo per-head learned sink logit is added to the attention softmax denominator anywhere in the eager or MLA attention implementations. | code → |
| Inkling | not in its code | not stated | verdict and linesNo learned per-head sink logit is added to the attention softmax denominator anywhere in the attention or kernel code. | code → |
| Hy3 | not in its code | not stated | verdict and linesThe attention forward computes a standard softmax over query-key scores with no extra learned sink logit column. | code → |
| GLM-5.2 | not in its code | not stated | verdict and linesNo learned per-head sink logit is added to the softmax denominator anywhere in the attention or indexer code. | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | not in its code | not stated | verdict and linesNo learned per-head sink logit is added to attention weights in eager_attention_forward or elsewhere; the softmax denominator is computed over only the key positions. | code → |
| MiniMax-M3 | not in its code | not stated | verdict and linesThe attention softmax has no extra learned sink logit; eager_attention_forward only uses the standard query/key/value matmuls. | code → |
| Step-3.7-Flash | not in its code | not stated | sink=falseThe config declares a 'sink' flag set to false and no per-head sink logit parameter or extra softmax column exists in the attention code. text_config.sink = false | code → |
| MolmoAct 2 | not in its code | not stated | verdict and linesNo learned per-head sink logit is added to the attention softmax denominator anywhere in the attention forward implementation. | code → |
| Qwen3.6-35B-A3B | not in its code | not stated | verdict and linesNo per-head learned sink logit is added to attention scores anywhere in the attention forward pass. | code → |
| Gemma 4 31B | not in its code | not stated | verdict and linesNo learned per-head sink logit is added to the attention softmax denominator anywhere in the attention implementation. | code → |
| Qwen3.5-397B-A17B | not in its code | not stated | verdict and linesNo learned per-head sink logit or extra softmax denominator column appears in the attention code. | code → |
| DeepSeek-V3.2 | not in its code | not stated | verdict and linesThe attention softmax implementation (eager_attention_forward) has no extra learned sink logit term added to the denominator. | code → |
| X-VLA | not in its code | not stated | verdict and linesNone of the attention modules (Florence2Attention, Florence2SdpaAttention, Florence2FlashAttention2, or the transformer.py Attention) add a learned per-head sink logit to the softmax denominator. | code → |
| OpenVLA / OpenVLA-OFT | not in its code | not stated | verdict and linesNo learned per-head sink logit mechanism is implemented anywhere in the provided modeling code. | code → |
In this branch 4
Everything filed at this node or below it, with one collapsible heading per child node.
filed here 4
By model
Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.
| Model | methods |
|---|---|
| DeepSeek-V4-Flash used | Attention sink used— |
| DeepSeek-V4-Pro used | Attention sink used— |
| MiMo-V2.5 used | Learnable attention sink bias used— |
| MiMo-V2.5-Pro used | Learnable attention sink bias used— |
| MiniMax-M3 not used | Learnable attention sink token not usedRoPE with attention sink unclear— |
| Xiaomi-Robotics-1 used | Learnable attention sink token used— |