LFM3-1.7B
Liquid AIA sigmoid gate scales each head’s output, so a head can switch off when it has nothing to add.
1 more from this model
a sigmoid gate scales each head's output, so a head can switch off when it has nothing to add
specific method · filed under model architecture
Scales each attention head's output with a sigmoid gate, allowing a head to be suppressed.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See normalization & residual for the methods filed beside it.
2 quotes from 2 sources, one card per model, each opening with its strongest.
A sigmoid gate scales each head’s output, so a head can switch off when it has nothing to add.
a sigmoid gate scales each head's output, so a head can switch off when it has nothing to add