specific method · filed under model architecture
Gated short convolution
A gated short convolution is a token-mixing block that applies input-dependent multiplicative gating around a depthwise short convolution along the sequence.1 It provides fast local mixing, and once a few attention blocks handle long-range retrieval, it can replace linear-attention or SSM branches.2,3
Trade-off It is inexpensive with good cache behavior, but needs a minority of attention blocks for long-range retrieval.4
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
each gated short convolution block applies input-dependent multiplicative gating around a depthwise short convolution
— LFM2 Technical ReportThis operator provides fast local mixing with excellent cache behavior on CPUs
— LFM2 Technical ReportOnce a handful of GQA blocks are available to handle long-range retrieval, the inexpensive gated short convolution alone is sufficient
— LFM2 Technical Reporta majority of inexpensive, gated short convolution blocks interleaved with a minority of grouped-query attention (GQA) blocks
— LFM2 Technical Report
Also called Gated short-convolution layers, Gated short convolutions, Gated short convolutions for local mixing, Input-aware gated short convolution blocks.