Open Models Almanac
Methodsmodel architecturenormalization & residual

specific method · filed under model architecture

QK-Clip

QK-Clip records each head's largest attention logit during training and, when it exceeds a threshold, shrinks that head's query and key weights after the optimizer step so the logit scales down. Kimi K2 introduced it for a latent-attention model, where per-head keys are not formed at decode time and QK-Norm does not apply as written.

Written for the explainer Normalization and logit control.

Also called QK-Clip technique, QK clipping for activation control.

sources
3
models
4
labs adopt it
0
strongest
evaluated

Documented in

Evidence

3 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
evaluated 1not used 2

Kolibri-1

Aleph Alphadiscussed1 source

An adapted QK-Clip (Kimi Team 2025) for GQA with threshold 100 in place of QK normalisation also shows gradient-norm spikes.

evaluatedmodel architecturein KolibriAleph Alpha
↑