Open Models Almanac
Methodsmodel architecturetoken mixerhybrid layer stacking

specific method · filed under model architecture

Hybrid Mamba-2 and Mixture-of-Experts Architecture

A hybrid layer-stacking architecture that interleaves Mamba-2 and Mixture-of-Experts layers, with selective Attention layers.1 It combines these token-mixer types in one model rather than using a single mixer throughout the network.2

How this definition was written

Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.

  1. The model employs a hybrid Mixture-of-Experts architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers — nemotron-3.5-lightning-30b-a3b Model by NVIDIA | NVIDIA NIM
  2. utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers — nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 · Hugging Face

Also called Hybrid Mixture-of-Experts architecture, Interleaved Mamba-2 and MoE layers, Selective attention layers, Selective Attention, select Attention layers.

sources
3
model
1
lab adopt it
1
strongest
core

Documented in

Evidence

6 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
core 6

The model employs a hybrid Mixture-of-Experts architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers.

coremodel architecturein Nemotron 3.5 LightningNVIDIA
5 more from this model

The model employs a hybrid Mixture-of-Experts architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers.

coremodel architecturein Nemotron 3.5 LightningNVIDIA

The model employs a hybrid Mixture-of-Experts architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers.

coremodel architecturein Nemotron 3.5 LightningNVIDIA

The model employs a hybrid Mixture-of-Experts architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers.

coremodel architecturein Nemotron 3.5 LightningNVIDIA

preserving selected Attention layers where global token interaction remains valuable

coreunclearin Nemotron 3.5 LightningNVIDIA

NVIDIA describes it as a hybrid: Mamba-2 + Mixture-of-Experts + Selective Attention + Multi-Token Prediction

coreunclearin Nemotron 3.5 LightningNVIDIA
↑