Open Models Almanac
Methodsmodel architecturetoken mixerlinear attention & state spaceMamba

specific method · filed under model architecture

Mamba

Mamba is a token-mixing architecture that, during autoregressive decoding, stores its SSM state in a constant-sized cache per batch element.1 Unlike attention's KV cache, this state does not grow with sequence length, though it can still exceed the KV cache at short sequence lengths.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. In Mamba autoregressive decoding, the SSM state is stored in a constant-sized cache for each batch element — Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
  2. The KV cache size, on the other hand, grows with sequence length. However, there is a crossover point below which the Mamba cache size is larger than the KV cache size — Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
sources
2
model
1
lab adopt it
1
strongest
core

Documented in

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
core 2

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAadopted2 sources

Nemotron 3 Ultra : Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

coremodel architecturein Nemotron 3 UltraNVIDIA
1 more from this model

Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

coremodel architecturein Nemotron 3 UltraNVIDIA
↑