Qwen3.5-397B-A17B
Alibaba (Qwen)integrates a linear attention mechanism with a sparse mixture-of-experts model
coremodel architecturein Qwen3.5-397B-A17BQwen Team
general family · filed under model architecture
A general attention family used as an efficient alternative to softmax attention, including in hybrid architectures.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See linear attention & state space for the methods filed beside it.
Also called linear attention mechanism.
2 quotes from 2 sources, one card per model, each opening with its strongest.
integrates a linear attention mechanism with a sparse mixture-of-experts model
Hybrid architectures replace a subset of softmax attention layers with efficient alternatives such as linear attention