Kolibri-1
Aleph AlphaAn adapted QK-Clip (Kimi Team 2025) for GQA with threshold 100 in place of QK normalisation also shows gradient-norm spikes.
specific method · filed under model architecture
QK-Clip records each head's largest attention logit during training and, when it exceeds a threshold, shrinks that head's query and key weights after the optimizer step so the logit scales down. Kimi K2 introduced it for a latent-attention model, where per-head keys are not formed at decode time and QK-Norm does not apply as written.
Written for the explainer Normalization and logit control.
Also called QK-Clip technique, QK clipping for activation control.
3 quotes from 3 sources, one card per model, each opening with its strongest.
An adapted QK-Clip (Kimi Team 2025) for GQA with threshold 100 in place of QK normalisation also shows gradient-norm spikes.
This allows stable training without explicit activation control such as qk-clip
we do not employ the QK-Clip technique