Open Models Almanac

Token mixer · T6

Attention sinks and output gates

A softmax head's attention always adds up to one: a learned sink logit lets it add up to less for one number per head, an output gate scales the head's result for a small projection, and the few direct comparisons are small and mixed

Illustration of a row of small balance scales on a long table. Each scale has a shallow bowl beside it into which a thin stream runs off, and one scale has a small hand-operated valve on its pan.

A transformer generating text stores the keys and values of past tokens in a key-value (KV) cache, so that each new token need not recompute them. In 2023, Xiao and colleagues limited that cache to the latest 1,024 tokens, evicting older ones, while Llama-2-13B read a 65,000-token book, and measured perplexity. It came out at 5,158: the model had stopped predicting. They then kept the first four tokens beside the latest 1,020, and perplexity fell to 5.40. Replacing those four tokens with newline characters still gave 5.60, so their position mattered and their content hardly did.

Heads in trained models put a large share of their attention on a few early tokens, relevant or not. Xiao et al. named these tokens attention sinks and traced them to the softmax: its weights must add up to one, so a head that finds nothing useful still has to put its attention somewhere.

Many of the models this site tracks use one of two cheap changes. A learned sink logit gives each head an entry that takes attention and contributes nothing. An output gate multiplies the head’s result by a learned factor, usually between 0 and 1, computed from the current token. In the site’s code checks, each model’s code tested against a checklist, 13 of 39 models have the sink (from four labs) and 9 have the gate (from six labs). One model has both and 18 have neither (one of them unclear on both).

Why the weights add up to one

In one attention head, the current token supplies a query qq and every token in view supplies a key kjk_j and a value vjv_j, each of width dhd_h (T1). The logit aj=q⊤kj/dha_j = q^\top k_j/\sqrt{d_h} scores token jj, and the head returns

o=∑jαjvj,αj=eaj∑meam.o = \sum_j \alpha_j v_j, \qquad \alpha_j = \frac{e^{a_j}}{\sum_m e^{a_m}} .

An output projection WOW_O then maps the concatenated heads back to the model’s width dd.

The weights αj\alpha_j always add up to one, and adding the same number to every logit leaves them unchanged, because the common factor cancels. So the logits’ overall level cannot tell the output that nothing matched. Take a toy head with four logits −3,−4,−4,−5-3, -4, -4, -5: its weights are 0.534, 0.197, 0.197 and 0.072. Add 6 to every logit, so that all four keys match far better, and the weights are identical (computed).

The output oo always lies in the convex hull of the values in view. A head that wants to pass on nothing has to find a mixture near zero, and the simple way is to put nearly all its attention on a token whose value is small. Training finds that option.

Figure 1.One attention head, with the places this article's fixes act. Stored keys and values are hatched. Gate positions are marked on the values, after the sum and after W_O; the sink logit is the marked column inside the softmax.

Sinks in trained models

The early tokens are the natural place. Every later token can attend to an early one, while a late token is seen by few, which Xiao et al. offer as the reason early tokens are easiest to train into a dumping ground. They saw Llama-2-7B heavily attending to the first token in all layers above the bottom two.

Later work looks at what sits in those positions. Sun et al. (2024) found massive activations in the residual stream of some early tokens: a few scalars of a hidden state above 100 in magnitude and at least about 1,000 times the median, above 15,000 in LLaMA2-70B. Attention concentrates on the tokens that carry them. Zeroing them in LLaMA2-7B made perplexity infinite, while replacing them with their means left it at 5.47, the original value on WikiText, so Sun et al. read them as “fixed but important biases”. Gu et al. (2024) looked at the keys and values projected from the first token and found smaller norms than other tokens’, and conclude the token is “storing extra attention and meanwhile not contributing to the value computation”. Huge hidden state and small projections can coexist because keys and values are projections of that state. Barbero et al. (2025) read the same behaviour as useful: an approximate no-op that lets a head avoid mixing.

So the literature offers two readings, a constant bias and a near no-op. A sink logit or a gate provides only the second. Sun’s zeroing shows the bias mattered in those models, but on the small models tested so far it is replaceable: in Gu et al.’s model, keys with a zero value gave the same loss (3.72 against 3.73). We found no controlled test at large scale.

Keeping the sink inside a token has two documented costs.

The first is eviction. Xiao’s Llama-2 was trained with full attention and windowed only at inference. When the window drops the sink token, its large term leaves the softmax denominator and the head’s remaining weights rise, in Xiao et al.’s words a “significant shift”. The size varies by model. With four initial tokens kept against none kept, on 400K tokens of text, Llama-2-7B went from 9.59 to 3,360 (window 4,096) and MPT-7B from 15 to 460 (window 2,048), while Falcon-7B only went from 12 to 18 and Pythia-12B from 12 to 22. A model trained with its window faces a different question, since its first token is out of view for most queries during training. On gpt-oss-20b, which also has a sink logit, the sliding-window layers put 0.3% of the non-sink attention on position 0 (Wang, 2026, preprint).

0%50%100%97%token 0a = 3v = 0.051.8%token 1a = -1v = 11.1%token 2a = -1.5v = 0.80.7%token 3a = -2v = -0.2sinknoneShare of the head's attentionWhat the head writes (output)0.07300.510%50%100%97%token 0logit 3value 0.051.8%token 1logit -1value 11.1%token 2logit -1.5value 0.80.7%token 3logit -2value -0.2sinknoneShare of the head's attentionWhat the head writes (output)0.07300.51

Token 0 holds 97% of the attention and adds almost nothing, so the head writes 0.073.

Toy. Token 0's logit and s are both 3.

Figure 2.A toy head in a model windowed only at inference. When the window evicts the sink token, its 97% of the attention is shared among the other three, and the head's output goes from 0.07 to 0.71. In the third mode a sink logit takes the same attention without a token; the output is lower than 0.07 because the sink adds no value of its own.

The second is outliers. Bondarenko et al. (2023) argue that a head trying to approach zero output must push its logit on a low-value token ever higher, which produces extreme activations; a few huge values then stretch a fixed-point range and leave ordinary values too few steps. On OPT-125m with 8-bit weights and activations, perplexity went from 15.84 at full precision to 21.18, and their gate reduced that loss to 16.02. These are small models from 2023 with fixed-point post-training quantization, so the result does not carry over directly to the floating-point formats in use today (I1). We found no open-model report that ties a sink to quantization failure. Liquid’s LFM3 blog says a head-wise gate “can help suppress activation outliers that hurt quantization”, with no number.

Keeping the sink tokens

StreamingLLM keeps the first few tokens’ KV entries in the cache forever and evicts from the middle, four entries per layer, with positions renumbered to cache positions so that RoPE sees familiar distances (T4). Xiao et al. found four tokens sufficed. The fix works only for sinks the model already built in those tokens, and they must stay in the cache.

They also compared this with two training changes on a retrained Pythia-160M, a model much less dependent on sinks than Llama-2 (perplexity 27.87 with no initial tokens cached, 18.05 with four):

Cached initial tokens014
Vanilla softmax27.8718.4918.05
Softmax with a zero entry29,21419.9018.01
Trained with a sink token at position 01,23518.0118.02

Neither change removes the need to keep tokens, and the sink-token model does best with one. The cause of the zero-entry blow-up is not given.

A sink logit: an extra column

A zero entry in the softmax is an old device. PyTorch’s add_zero_attn option appends an all-zero key and value, and Merity et al. (2016) added a learned attention column whose mass went to a separate source. Evan Miller’s 2023 post, on quantization outliers, proposed it as softmax1, adding 1 to the denominator so a head can output nearly nothing. gpt-oss made the extra logit learned, one per head: its card says each head “has a learned bias in the denominator of the softmax … which enables the attention mechanism to pay no attention to any tokens”. For one head with sink logit ss, write S=∑meamS = \sum_m e^{a_m}:

o=∑jeajvjes+S=τ∑jαjvj,τ=Ses+S=σ(log⁡S−s),o = \frac{\sum_j e^{a_j} v_j}{e^{s} + S} = \tau \sum_j \alpha_j v_j, \qquad \tau = \frac{S}{e^{s}+S} = \sigma(\log S - s),

where σ\sigma is the sigmoid and αj\alpha_j is still the ordinary softmax. The ratios among real tokens are unchanged, and the sink multiplies the whole output by τ\tau. A large ss takes more attention, and s→−∞s \to -\infty recovers plain softmax. Softmax1 is s=0s = 0.

In the code the sink really is a column. gpt-oss’s torch.cat([attn_weights, sinks], dim=-1) appends it after the window mask, a softmax runs over the combined logits, and probs[..., :-1] drops it. Nothing is added in place of the dropped mass, since the sink has no value. The same code runs in training and in decode, the phase that generates one token at a time.

Return to the toy head at s=0s = 0. With its weak logits, τ=0.085\tau = 0.085: the head writes 8.5% of its usual output. Shift all logits up by 6 and τ=0.974\tau = 0.974, with the same weights among the four tokens (computed).

  • attention on a token
  • its share among the tokens
  • sink column
Attention by token0.00.51.00.534a = -3.00.197a = -4.00.197a = -4.00.072a = -5.00.915s = 0.0Kept mass τ (how much the head writes)0.085plain softmax = 10Attention by token0.00.51.00.534logit -3.00.197logit -4.00.197logit -4.00.072logit -5.00.915sink, s 0.0Kept mass τ (how much the head writes)0.085plain softmax = 10

Kept mass 0.085: the head writes 8.5% of its usual output. Token 0 is still 2.72 times token 1.

Figure 3.A toy head. Without a sink, moving every logit by the same amount changes nothing. With one, the same move changes only how much the head writes: 8.5% at the low setting and 97% six logits higher.

τ\tau does not read match quality alone. log⁡S\log S lies between the largest logit and that logit plus log⁡n\log n for nn tokens in view, so more tokens raise it too: with every logit at −3-3 and s=0s = 0, τ\tau is 0.047 for one token, 0.86 for 128 and 0.998 for 8,192 (computed). A global layer’s view grows with position and a window’s stays fixed, so one constant ss sets different abstention levels at different positions unless the keys and queries offset it (our inference). Wang measured 48% of attention on the sink in gpt-oss-20b’s sliding-window layers and 34% in its full-attention layers, in the direction this predicts, on eight blocks of 1,024 tokens.

What it costs

It costs one scalar per head in every layer that carries it. gpt-oss-120b has 36 layers of 64 heads, so 2,304 numbers (computed from the config and the code’s parameter shape); DeepSeek-V4-Pro has 61 layers of 128 heads, 7,808 numbers, about 5 parts per billion of its parameters, against our own bottom-up count of about 1.57 trillion. It stores nothing per token, so it adds no bytes to the KV cache, and prefix caching is unaffected.

The engineering cost is that the column needs kernel support. Attention libraries ship a fused routine, which computes the weighted sum without forming the weights, and PyTorch’s scaled-dot-product routine (SDPA) has no sink term, so gpt-oss disables it and MiMo’s code falls back to a slower eager path when SDPA is requested. FlashAttention-style backends take the sink as an argument, and vLLM’s kernels for DeepSeek-V4.1 have a sinks= input. Fu et al. (2026, preprint) point out a route that needs only the log-sum-exp log⁡S\log S, which fused kernels already return: multiply the output by τ\tau afterwards. Two ports run only eagerly: DeepSeek-V4’s Transformers code turns off FlashAttention, SDPA and FlexAttention, and Hy4-preview’s says “Combination of sinks and DSA disable anything but eager atm”.

What trains it

The ordinary loss trains it: with y=∑jαjvjy = \sum_j \alpha_j v_j, σ′=σ(1−σ)\sigma' = \sigma(1-\sigma) gives ∂o/∂s=−τ(1−τ) y\partial o/\partial s = -\tau(1-\tau)\,y. The factor τ(1−τ)\tau(1-\tau) peaks at τ=1/2\tau = 1/2 and vanishes on both sides for one query, though ss is shared by every query of the head and log⁡S\log S varies across them. In the Transformers code a fresh sink starts at zero for DeepSeek-V4 and Hy4-preview and as a small normal draw for gpt-oss; the labs’ training values are not stated. Near-zero logits with s≈0s \approx 0 give τ=n/(n+1)\tau = n/(n+1), so in a long context the sink starts nearly unused. Trained values can move far from there: Wang reports gpt-oss-20b’s sink logits averaging 1.1 to 4.1 per layer. A learned ss is the same as giving the head’s logits a learned offset, which a dot product of a query and a key lacks.

MiMo’s released code sets requires_grad=False on its sink. Whether the value was fixed or trained earlier cannot be told from the code.

When it helps

  • Softmax1 has two published negatives: the table above, where the zero-entry model still needs initial tokens, and Sun et al.’s GPT-2, where it did not remove massive activations.
  • MiMo-V2-Flash ablated a learned sink on a 32B dense test model pre-trained at 8,192 sequence length, with window 128 and a test-model ratio of sliding-window to global layers it does not state. On MMLU, a benchmark of general knowledge, the hybrid scored 54.9 without a sink and 58.3 with one, against 57.3 for global attention throughout, and GSM8K did not move (36.9 either way). Its introduction credits the sink for the shipped 128-token window at five sliding-window layers per global one. The long-context tables have a sink in every hybrid arm, so that credit is untested at long context.
  • MiniMax tested a gpt-oss-style sink on a 10B pilot of its sparse-attention model and left it out. It “does not completely remove the original first-token sink” and “does not yield a clear or consistent improvement” on perplexity, in a figure only. That model has no sliding window, and its selector consistently selects the first key block.
  • DeepSeek-V4’s report states the sink’s purpose (“even to be near 0”) and gives no measurement of its effect.

With the sink logit in trained models, attention does move to it. Wang measured 12% of attention on the sink in a 10M model and 54% at 350M, and first-token attention falling below 1% in his 124M and 350M models, with the first token’s value norm back to that of other tokens. So a sink logit changes what training learns, even though at fixed logits it preserves the ratios. Wang’s six full-attention layers of gpt-oss-20b that kept 9% to 17% on position 0 read it as content, at or above the usual value norm, so sink-logit models need not abandon early tokens.

An output gate

Gating an output by a sigmoid of a learned projection is an old device: gated linear units (Dauphin et al., 2016) and Squeeze-and-Excitation (Hu et al., 2017) both do it. Applied to a head, the gate multiplies the attention result before the output projection:

o=γ(x)⊙∑jαjvj,γ(x)=σ(Wγx),o = \gamma(x) \odot \sum_j \alpha_j v_j, \qquad \gamma(x) = \sigma(W_\gamma x),

where xx is the current token’s hidden state after the layer’s input norm, WγW_\gamma is a learned linear map and ⊙\odot multiplies element by element (a scalar in the head-wise case). Bondarenko et al. applied this to transformer attention heads in 2023, to remove outliers, computing the gate from a head-sized part of the input. On OPT-125m the largest activation fell from 340 to 8.7. Qiu et al. (2025) reintroduced it for language models from the whole hidden state.

Gates differ along two axes. Width is one number per head or one per element of the head’s output. Sharing is each head with its own gate or all heads sharing one. Models in the map use two widths: one per head (Step-3.7-Flash, Step-5-Preview, LFM3-1.7B, Laguna-S-2.1) and one per element (Qwen3.5, 3.6 and 3.8, Kimi K3’s latent-attention layers, Hy4-preview). All use sigmoids except Laguna, whose softplus has no upper bound and can amplify a head, and each has a separate gate per head.

A gate acts after the softmax and leaves every weight alone, yet first-token attention falls in models trained with one. Our reading: a head can silence itself by driving γ\gamma towards 0, so training no longer has a reason to concentrate attention on a low-value token. The softmax is unchanged, and the learned logits change. Qiu et al. note that gated hidden states look like baseline ones and suggest the gate may serve a similar function, and they do not establish causality.

What trains it

For a head with gate γ\gamma, ∂o/∂z=γ(1−γ) y\partial o/\partial z = \gamma(1-\gamma)\,y, where zz is the gate’s input and yy the head’s output, so a gate near 0 or 1 stops learning for that token, as the sink does. Every parameter upstream of the gate gets gradient scaled by γ\gamma. A head near 0 is cut off from its own training signal, and Qiu’s mean gate score of 0.116 in their per-element form suggests the case is common (our inference). Bondarenko et al. found the initial gate level matters too: useful starting scores of about 0.25 to 0.9 for BERT, with strong degradation when very low. The deployed gates have no bias term, so γ=1/2\gamma = 1/2 at x=0x = 0.

No fix removes the need for a large number. To reduce a head’s output to a fraction ϵ\epsilon, a real sink token needs a logit margin of log⁡((1−ϵ)/ϵ)\log((1-\epsilon)/\epsilon), a sink logit needs that margin over log⁡S\log S, and a gate needs WγxW_\gamma x at that margin below zero: 4.6 at ϵ=0.01\epsilon = 0.01 in all three cases (computed). The fixes place that margin in a parameter or a dedicated map, instead of in activations shared with the attention pattern.

What it did in the one study that measured it

Qiu et al. trained a 15B model with 2.54B active parameters (C2) on 400B tokens with the gate at several positions. The chart below shows every row of their table. “Output gate” means multiplying after the attention sum, computed from the current token; “gate on values” scales each value before the sum; “shared across heads” uses one gate for all heads; “ignoring the input” has fixed learned values; “kept above 0.5” uses a sigmoid squeezed into 0.5 to 1.

Figure 4.One study, one model size (15B parameters, 2.54B active). Only the two head-specific output gates with a plain sigmoid bring first-token attention below 10%. The gate on values has activations almost as low (125), yet its first-token attention is 0.30.

Our reading of why the other rows fail: a shared gate cannot silence one head alone; a gate ignoring the input cannot decide per token; a gate above 0.5 can only halve a head. A gate on values cuts activations almost as much as the output gate but leaves first-token attention at 0.30, which suggests massive activations “are not a prerequisite for attention sinks”. The Qiu baseline’s perplexity was 6.03 averaged over their benchmark set, and the best gate reached 5.76. Clearing the sink and gaining perplexity come apart: the gate kept above 0.5 reaches 5.90 and the gate on values 5.82 while most of the sink stays.

Two other findings hold within narrow limits. A 48-layer 1.7B baseline, trained at their highest learning rate 8e-3 for 400B tokens, reached perplexity 9.2, and the same model at batch 4,096 for 1T tokens diverged, while the gated versions trained at 7.3 and 7.1. Clipping activations at 100 or 300 did not rescue the baseline, so the authors say the instability “is not solely due to large activations”. On retrieval, models trained on 3.5T tokens then 80B tokens at 32K, with positions stretched to 128K by YaRN (T4), scored 37.9 without the gate and 72.9 with it on RULER at 32K, a retrieval benchmark scored 0 to 100. Without the stretch they made almost no difference at 32K (79.5 against 79.8). Qiu et al. suggest as a hypothesis that baseline models rely on sinks whose pattern fails to adapt when the context is stretched.

What it costs

The gate is a linear map from dd to its width, d×d \times width parameters per layer, and it stores nothing per token. Qwen3.5-397B-A17B has d=4,096d = 4{,}096, 32 heads of width 256 and 15 gated layers: 4,096×8,192×15=5034{,}096 \times 8{,}192 \times 15 = 503 million parameters, about 0.13% of the model (computed; the code doubles the query projection). A head-wise gate is smaller by the head width: Step-3.7-Flash has 16 million, where one per element would be 2.1 billion. The largest in the map is Hy4-preview’s 7.85B, about 1% of the model. The chart after the model tables shows every softmax gate.

The cost depends on the phase. In prefill, running the prompt at once and limited by arithmetic, the gate adds about 503M multiply-adds per token for Qwen3.5. In decode, limited by bytes read per step, the gate’s weights are read once per step for the whole batch. Of the 17B parameters one Qwen3.5 token activates, they are 3%, which applies only at batch 1. At batch 64 they are about 0.2% of weights read per step (computed, assuming uniform routing among 512 experts with 10 chosen), and the 15 gated layers’ cache alone is about 4.0B numbers per request at 256K tokens, eight times the whole gate.

Qiu et al. report under 2% added wall time without naming the phase, and Bondarenko et al. 3% to 8% extra pre-training compute for their linear gate. Neither is stated as a decode measurement, and we found no source measuring either mechanism’s decode latency. No document discusses the kernel for the attention gate, though vLLM’s Step-5 code has a dedicated attention_gate operation. NVIDIA’s released DSpark drafter (P1) uses per-head attention sinks in every layer, the only drafter whose documents name a sink.

Low precision

The Transformers code keeps DeepSeek-V4’s and Hy4-preview’s sink logits in FP32, and in all three ports re-centres the combined logits, an addition it says avoids BF16 overflow. Near-zero gates carry a risk: Nemotron 3.5’s report warns that an SSM gate emits values that flush to zero under FP4 activation quantization, and a per-element attention gate with a low mean score could meet the same problem (our inference). Kimi K3 keeps its attention output in FP32 during training, directly beside its gate, for a different reason, the biased rounding error of flash attention.

What the gate does not do

The argument for a gate’s non-linearity runs as follows. The values reach the output through WVW_V and WOW_O, with WVW_V mapping a token’s state to the head’s value and WOW_O back to width dd, and for fixed attention weights the map through one head has rank at most dhd_h. A gate inserts a diagonal D=diag(γ)D = \text{diag}(\gamma), giving WODWVW_O D W_V, and rank at most dhd_h still holds (our derivation). The nonlinearity sits in the query: for a fixed current token, the output stays linear in the values.

Qiu’s table points to what the gate buys. A gate ignoring the input, exactly a reparametrized WOW_O and no new function, still improved perplexity (5.92 against 6.03); that is an optimisation or implicit regularisation effect, and we read the row that way. A per-head RMSNorm with almost no parameters reached 5.85, a gate above 0.5 with dependence on the current token but no low scores reached 5.90, and the output gates with low scores reached 5.79 and 5.76. These differences are small, and the paper gives no uncertainty.

A sink and a gate side by side

Written together, the two are close. A sink logit multiplies the head’s output by τ=σ(log⁡S−s)\tau = \sigma(\log S - s), and a head-wise gate multiplies it by σ(Wγx)\sigma(W_\gamma x). So a sink is an implicit head-wise gate, as Fu et al. write out.

Figure 5.A toy head. The sink's scale follows the logits in view, and the gate's follows the current token (illustrative value here), which may carry what earlier layers read. A per-element gate can scale coordinates differently, and a single scalar cannot.

The sink reads the logits in view, and also the current token, because a component shared by all keys makes logits depend on it (a possibility, untested in trained models). A query-only gate reads the current token and can scale elements differently. Neither contains the other. Neither changes the ratios among tokens in view at given logits, and neither gives a non-zero bias the way a real token’s value does.

One study compares a sink with a gate computed from the current token. Fu et al. trained 0.6B, 1B and 2B models from scratch with no mechanism, a sink, and a head-wise gate. Their text says the gate does best. In the table’s base rows, on average accuracy over ten tasks, the sink is ahead of the gate at 1B and 2B (43.27 against 42.69, 46.35 against 46.30), and the gate leads at all three sizes only with their auxiliary balancing loss, which itself moves accuracy by as much as the gaps. Wang compares a sink logit with gates computed from values at 10M to 124M, and sink plus gate against the sink alone up to 350M: combining them improved validation loss by 0.0089 ± 0.0027 nats over the sink at 350M, while the sink’s own gain over the baseline there (0.0103) is not resolved beyond seed spread. We have not read Bu et al. (2025), which Wang cites as finding a value gate ahead of a learnable sink at 125M.

None of these compares a sink with a gate in a model trained with a window, the setting most deployed models use.

Gates on linear mixers

A linear mixer (T2) has no softmax, and the same gate appears there. Gated DeltaNet and KDA compute a recurrent output, apply a per-head RMSNorm and then gate it: Qwen3.8-Next writes ot=Wo[σ(Wzxt)⊙RMSNorm(yt)]o_t = W_o[\sigma(W_z x_t) \odot \text{RMSNorm}(y_t)], with WzW_z in the role of WγW_\gamma. In Qwen3.5 and 3.6 these linear-mixer layers use SiLU, which can be negative and has no upper bound, and in Qwen3.8 they use a sigmoid. The Qwen3.8-Next report says it “observe[s] consistent improvements” with a sigmoid, and gives no number. Qwen’s softmax layers use a sigmoid throughout.

“Low rank” here means factoring WγW_\gamma through a narrow bottleneck, a different thing from the rank above. Kimi K3 replaces the low-rank KDA gate of Kimi Linear with a full-rank one and adds a full-rank gate to its MLA layers, with no ablation of either. GLM-5.3-Flash keeps a low-rank KDA gate through width 128, 21 times smaller than a full-rank one of its width (computed). Nemotron’s Mamba-2 layers apply the gate before the group RMSNorm, with SiLU.

Linear-mixer gates add to the softmax gates’ parameter cost: Qwen3.5’s GDN gate is 1.51B, so its gates together are 2.0B, 12% of the 17B active.

Comparison

Keep sink tokensSoftmax1 (s=0s=0)Sink logitHead-wise gatePer-element gate
Changeswhat the cache holdssoftmax denominatorsoftmax denominatorhead’s output, one scalehead’s output, per element
Readspositionlogits in viewlogits in viewcurrent tokencurrent token
Parametersnonenonehh per layerd×hd \times h per layerd×hdhd \times h d_h per layer
KV-cache bytes4 entries per layernonenonenonenone
Kernelnonean extra columnan extra column, or multiply by τ\tauprojection and scaleprojection and scale
Direct evidenceXiao: restores perplexity after evictiontwo negativesMiMo on general benchmarks; MiniMax negativeQiu, FuQiu
Weaknessa sink token must exist and stayfixed level; negativesτ\tau depends on token countno per-element shapinglargest parameter cost

Where the models in the map sit

A code check tests each model’s code against a checklist; 39 models have a usable row. In the learnable-attention-sink check, 13 are present. Eleven have the extra column in their code, and for the two DeepSeek-V4.1 rows we saw only the kernel call. Coverage differs:

ModelLayers carrying the sinkStated in documents?
gpt-oss-120ball 36, full and slidingyes, card
DeepSeek-V4 Flash, Pro, 0731, 0813, vision-expevery layer (one class for sliding and compressed); report says CSA and HCAyes, §2.3.3
DeepSeek-V4.1-Flash and -1017seen at the kernel call onlyno
MiMo-V2.5, V2.6-Flashsliding-window layers: 39 of 48yes, V2-Flash report (V2.6 by reference)
MiMo-V2.5-Pro, V2.6-Prosliding-window layers: 60 of 70yes, V2-Flash report (V2.6 by reference)
Hy4-previewall 78config only

MiMo’s report says 5:1 with a global first layer, so 39 to 9 in its Table 1 and the configs; the Pro models’ card says 6:1, and their configs 60 to 10. In gated-attention, 9 models are present, each computed from the current token:

ModelWidthActivationGated layers
Qwen3.5-397B-A17Bper elementsigmoid15 of 60 (others linear)
Qwen3.6-35B-A3Bper elementsigmoid10 of 40 (others linear)
Qwen3.8-Flash-Nextper elementsigmoid12 of 48 (full_attention in config, sparse in the card)
Kimi K3per elementsigmoid24 MLA layers
Hy4-previewper elementsigmoidall 78
Step-3.7-Flash, Step-5-Previewper headsigmoidall attention layers
LFM3-1.7Bper headsigmoid11 of 24
Laguna-S-2.1per headsoftplusall 48

Only Hy4-preview has both, on every layer, and its documents give no reason: they call its attention “Gated DSA” and do not mention a sink. Elsewhere a model has at most one. Six labs use the gate and four the sink (Tencent is in both), and 18 of the 39 models have neither. The check is about softmax output, so GLM-5.3-Flash and two Nemotron models, which gate linear mixers, count as absent. Three gate rows and one sink row are unclear because their code was not available. T2 noted a pattern in window sizes: the three labs at 128 tokens use a sink logit, and two of the five labs at 512 or more (Poolside, StepFun) gate the output. The check counts those as separate choices.

A config field is not evidence of a mechanism. Fields such as Qwen’s attn_output_gate, Hy4’s gated_mla and learnable_sink, and MiniMax-M3’s attention_output_gate: false are not read by the code in the pinned files; the last model has no gate at all in its modeling code.

Figure 6.Parameter counts computed from each model's config and the code's parameter shapes, on a log axis, softmax gates only. Three of the 13 sink rows are shown, including the smallest and the largest; they differ by a factor of about 3, while gates differ by more than 20,000 times.

Easy to get wrong

  • A sink logit and a sink token are different things. A token has a value, can add a bias to the output, and can be evicted. A logit has no value and is a parameter of the head. A logit gives a head a place to put attention that a window cannot evict; whether training then stops relying on early tokens is an empirical matter, and the studies differ.
  • The sign: a larger ss takes more attention, a very negative one is ordinary softmax, and softmax1 is s=0s = 0.
  • “Gating removes sinks” is true only of gates that read the current token, belong to one head and use a plain sigmoid, in the one study that tested this, and Laguna’s softplus was not among those tested.
  • The drop in loss spikes from 32.0 to 3.2 per 10,000 steps reported by Qwen3.8-Next belongs to GatedNorm on the residual stream (R1), not to attention gates.
  • MiMo’s window result, 128 beating 512, has a sink in both arms and says nothing about the sink. The sink’s ablation is on a 32B test model at 8,192 tokens.

Check yourself

  1. A head has logits 00 and −2-2, values 11 and 00, and sink logit s=0s = 0. What is its output, and what would it be without the sink?
    AnswerThe sum of exponentials is 1+e−2=1.1351 + e^{-2} = 1.135. With the sink the denominator is 2.1352.135, so o=1/2.135=0.468o = 1/2.135 = 0.468. Without the sink, o=1/1.135=0.881o = 1/1.135 = 0.881.
  2. Kimi K3 has d=7,168d = 7{,}168 and 24 gated MLA layers of 96 heads with value width 128. How many parameters does its attention gate have, and how many would a head-wise gate need?
    AnswerPer element: 7,168×(96×128)×24=2.117{,}168 \times (96 \times 128) \times 24 = 2.11 billion. Head-wise: 7,168×96×24=16.57{,}168 \times 96 \times 24 = 16.5 million, 128 times fewer.
  3. Why can a window not evict a sink logit, and what does the logit fail to guarantee?
    AnswerIt is a parameter of the head and joins the softmax after the window mask, so it is present at every step, whatever the window holds. It does not guarantee that training stops using early tokens as sinks: in several heads of MiniMax’s sparse-attention pilot the first token still received substantial attention.
  4. In Qiu et al.’s table, why would a gate kept above 0.5 leave the sink in place?
    AnswerOur reading: the sink forms because a head needs a way to write almost nothing. A gate limited to the range 0.5 to 1 can only halve the head’s output, so the pressure to dump attention on a low-value token remains. The table shows first-token attention of 0.451 for that gate against 0.048 for the plain sigmoid.

In the map

Methods

Taxonomy nodes

Further reading

  • Xiao et al. (2023), arXiv:2309.17453: StreamingLLM, the eviction experiments, and the comparison with a zero entry on a small retrained model.
  • Sun et al. (2024), arXiv:2402.17762: massive activations and explicit attention biases; §B.4 for the softmax1 negative.
  • Miller (2023), “Attention Is Off By One”: the softmax1 proposal.
  • Gu et al. (2024), arXiv:2410.10781, and Barbero et al. (2025), arXiv:2504.02732: when sinks emerge and what they do.
  • Bondarenko et al. (2023), arXiv:2306.12929: outliers, clipped softmax and a per-head gate on attention, for 8-bit quantization.
  • Qiu et al. (2025), arXiv:2505.06708: the gated-attention paper, its variants and the sink table.
  • Fu et al. (2026), arXiv:2602.01203, and Wang (2026), arXiv:2609.22005 (preprints): the sink as an implicit gate, and tests on small models.
  • MiMo-V2-Flash report (2026), arXiv:2601.02780, §2.2, and MiniMax Sparse Attention, arXiv:2606.13392, App. B.5: the positive and negative sink ablations in model reports.
↑