Xiaomi-Robotics-1
XiaomiWe add a learnable attention sink token at the front of the state and noisy action tokens to stabilize the attention distribution during training.
specific method · filed under model architecture
A learnable attention sink token gives each attention head an extra learnable sink logit to represent the attention-sink behavior that otherwise falls on the first token.1 The sink logit competes with normal key positions in the attention softmax, absorbing attention mass instead of the first real token acting as an implicit sink.2
Trade-off It adds parameters and implementation complexity and, in the reported test, did not yield a clear or consistent improvement, so it was not adopted.3
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
each attention head is given an additional learnable sink logit, which competes with normal key positions in the attention softmax— MiniMax Sparse Attention
which competes with normal key positions in the attention softmax— MiniMax Sparse Attention
Given its additional parameters, implementation complexity, and the fact that it does not fully suppress first-token sink behavior, we do not include the learnable attention sink in the final recipe— MiniMax Sparse Attention
Also called learnable attention sink, a learnable attention sink token.
2 quotes from 2 sources, one card per model, each opening with its strongest.
We add a learnable attention sink token at the front of the state and noisy action tokens to stabilize the attention distribution during training.
we do not include the learnable attention sink in the final recipe