specific method · filed under model architecture
Mamba-2, Mixture-of-Experts, and Selective Attention Hybrid
Combines Mamba-2 and mixture-of-experts layers with selected attention layers; the evidence also names multi-token prediction as part of one described hybrid.
Also called Hybrid Mixture-of-Experts architecture, Interleaved Mamba-2 and MoE layers, Selective attention layers, Selective Attention, select Attention layers.
- sources
- 4
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Evidence
7 spans quoted from the sources, strongest treatment first.
The model employs a hybrid Mixture-of-Experts architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers.
The model employs a hybrid Mixture-of-Experts architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers.
The model employs a hybrid Mixture-of-Experts architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers.
The model employs a hybrid Mixture-of-Experts architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers.
NVIDIA describes it as a hybrid: Mamba-2 + Mixture-of-Experts + Selective Attention + Multi-Token Prediction
preserving selected Attention layers where global token interaction remains valuable
Nemotron 3 models predominantly interleave MoE layers with cheaper Mamba-2 layers
Filed alongside
Other methods under model architecture :: token mixer :: hybrid layer stacking.