Open Models Almanac
Methodsmodel architecturetoken mixerhybrid layer stacking

general family · filed under model architecture

Hybrid Mamba-Transformer

Combines Mamba-family sequence-model layers with Transformer attention layers; the supplied mentions often specify a Mixture-of-Experts configuration but do not establish one exact layer pattern.

The sources on this page name it without explaining how it works. The line above is the extraction's one-line gloss. See hybrid layer stacking for the methods filed beside it.

Also called hybrid Mamba‑Transformer MoE, Mamba-Transformer.

sources
6
models
3
lab adopt it
1
strongest
core

Documented in

Evidence

9 quotes from 6 sources, one card per model, each opening with its strongest.

How sources treat it
used 2core 7

the Nemotron 3 family of models use a Mixture-of-Experts hybrid Mamba-Transformer architecture that pushes the accuracy-to-inference-throughput frontier

coremodel architecturein Nemotron 3 familyNVIDIA
3 more from this model

both models combine large-scale distillation, long-context training, and native speculative decoding

coremodel architecturein Nemotron 3.5 Super and Nemotron 3.5 LightningNVIDIA

powered by hybrid Mamba‑Transformer MoE with 1M-token context

coremodel architecturein Nemotron 3 familyNVIDIA

Nemotron 3 models predominantly interleave MoE layers with cheaper Mamba-2 layers

coremodel architecturein Nemotron 3 familyNVIDIA

open and efficient Mixture-of-Experts Hybrid Mamba-Transformer models designed for reasoning, long-context understanding, and agentic workloads

coremodel architecturein NVIDIA Nemotron 3.5 Super and NVIDIA Nemotron 3.5 LightningNVIDIA
1 more from this model

both models combine large-scale distillation, long-context training, and native speculative decoding

coremodel architecturein Nemotron 3.5 Super and Nemotron 3.5 LightningNVIDIA

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAadopted2 sources

Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

coremodel architecturein Nemotron 3 UltraNVIDIA
3 more from this model

Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

coremodel architecturein Nemotron 3 UltraNVIDIA

Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model

usedmodel architecturein Nemotron 3 UltraNVIDIA

Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model

usedmodel architecturein Nemotron 3 UltraNVIDIA
↑