Open Models Almanac
Methodsmodel architecturetoken mixerlinear attention & state spaceMamba

methods · linear attention & state space

Mamba

4 methods filed at this node or below it, from the sources of 3 models.

In practice

Checked as “Mamba-2 layers” against each model’s own code and config.json, apart from its documents.

✓in its code✕not in its code
ModelCodeIts documentsParametersLink
NVIDIA-Nemotron-3.5-Super-VL-120B-A12Bin its codenot stated
mamba_num_heads=128 · ssm_state_size=128 · mamba_head_dim=64 · chunk_size=128

The text_config declares Mamba-2-style SSM hyperparameters (ssm_state_size, mamba_num_heads, mamba_head_dim, chunk_size) used by the 'mamba' layers in layers_block_type, matching NemotronH's Mamba-2 mixer design.

llm_config.ssm_state_size = 128llm_config.mamba_num_heads = 128llm_config.mamba_head_dim = 64llm_config.chunk_size = 128
NVIDIA-Nemotron-3.5-Lightning-30B-A3Bin its codeadopted
mamba_num_heads=64 · ssm_state_size=128

The 'mamba' layers use NemotronHMamba2Mixer implementing the Mamba-2 SSD chunked scan with per-head scalar decay (A_log, D, dt_bias) and ssm_state_size.

mamba_num_heads = 64ssm_state_size = 128
self.A_log = nn.Parameter(torch.empty(self.num_heads))
        self.norm = NemotronHRMSNormGated(
            self.intermediate_size, group_size=self.intermediate_size // self.n_groups, eps=config.layer_norm_epsilon
        )
        self.D = nn.Parameter(torch.empty(self.num_heads))
modeling_nemotron_h.py · L440–444
NVIDIA-Nemotron-3-Ultra-550B-A55Bin its codeadopted
mamba_num_heads=256 · ssm_state_size=128

The linear-attention layers are implemented as NemotronHMamba2Mixer, a Mamba-2 selective state-space block with chunked SSD scan and per-head scalar decay (A_log/D), configured via mamba_num_heads and ssm_state_size.

mamba_num_heads = 256ssm_state_size = 128
class NemotronHMamba2Mixer(nn.Module):
modeling_nemotron_h.py · L389
self.A_log = nn.Parameter(torch.empty(self.num_heads))
modeling_nemotron_h.py · L440
Mistral-Large-4not in its codenot stated
verdict and lines

No Mamba-2/SSD state-space block implementation or related config keys (mamba_num_heads, ssm_state_size) are present.

LFM3-1.7Bnot in its codenot stated
verdict and lines

No Mamba-2 SSD selective state-space block is implemented; non-attention layers are short causal convolutions (Lfm3ShortConv), not SSM.

MiniMax-M3.1-Flashnot in its codenot stated
verdict and lines

No Mamba-2/SSD state-space block exists in the given modeling code.

DeepSeek-V4.1-Flash-1017not in its codenot stated
verdict and lines

No Mamba-2 SSD state-space blocks are implemented or referenced in the config or code.

Step-5-Previewnot in its codenot stated
verdict and lines

No Mamba-2 SSD blocks are implemented or referenced; the model is a GQA transformer with sparse attention.

Kolibri-1not in its codenot stated
verdict and lines

No SSM/Mamba-2 state-space block or related config keys (mamba_num_heads, ssm_state_size) appear in this model.

MiMo-V2.6-Flashnot in its codenot stated
verdict and lines

No Mamba-2/SSD selective state-space block is implemented anywhere in the given files.

MiMo-V2.6-Pronot in its codenot stated
verdict and lines

No Mamba-2/SSD selective state-space block is implemented anywhere in the modeling file.

DeepSeek-V4.1-Flashnot in its codenot stated
verdict and lines

No Mamba-2 SSD state-space block implementation is present in the provided code or config.

Hy4-previewnot in its codenot stated
verdict and lines

No Mamba-2/SSD state-space block implementation or related config keys (mamba_num_heads, ssm_state_size) are present.

GLM-5.3-Flashnot in its codenot stated
verdict and lines

No Mamba-2/SSD mechanism (no mamba_num_heads, ssm_state_size, or per-head scalar decay SSM module) exists; the recurrent mixer implemented is Kimi Delta Attention, not Mamba-2.

Qwen3.8-Flash-Nextnot in its codenot stated
verdict and lines

No Mamba-2/SSD selective state-space block is implemented; the recurrent mixer present is the gated delta rule (GatedDeltaNet), not Mamba-2.

DeepSeek-V4-Flash-Vision-Expnot in its codenot stated
verdict and lines

No Mamba-2 selective state-space block is implemented in this file; there is no ssm_state_size or mamba head config and no SSD code path.

GLM-5.3not in its codenot stated
verdict and lines

No Mamba-2/SSD selective state-space block code exists in the given files.

DeepSeek-V4-Pro-0813not in its codenot stated
verdict and lines

No Mamba-2 SSD selective state-space block is implemented in the modeling file

DeepSeek-V4-Flash-0731not in its codenot stated
verdict and lines

No Mamba-2 selective state-space blocks are implemented in the modeling file.

Xiaomi-Robotics-1not in its codenot stated
verdict and lines

No Mamba-2 SSD blocks or ssm_state_size/mamba_num_heads config exist; the model is a standard transformer decoder.

Laguna-S-2.1not in its codenot stated
verdict and lines

No Mamba-2 SSM block or mamba_num_heads/ssm_state_size config exists; all layers are LagunaAttention.

Kimi K3not in its codenot stated
verdict and lines

No Mamba-2/SSD selective state-space block (mamba_num_heads, ssm_state_size) exists in the code or config; the linear-attention layers use KDA (delta-rule), not Mamba-2.

Inklingnot in its codenot stated
verdict and lines

There are no Mamba-2 SSD blocks; config only carries short-conv (sconv) parameters, not SSM state sizes or per-head decay.

Hy3not in its codenot stated
verdict and lines

No SSM/Mamba-2 selective state-space block or related config keys (mamba_num_heads, ssm_state_size) appear in this model.

GLM-5.2not in its codenot stated
verdict and lines

No Mamba-2 SSD state-space blocks are implemented in the provided modeling file.

MiniMax-M3not in its codenot stated
verdict and lines

No Mamba-2 SSM blocks or related config keys (mamba_num_heads, ssm_state_size) are present.

Step-3.7-Flashnot in its codenot stated
verdict and lines

No Mamba-2 SSD block or mamba-related config keys (mamba_num_heads, ssm_state_size) are present in the code or config.

MolmoAct 2not in its codenot stated
verdict and lines

No Mamba-2/SSD selective state-space block implementation or related config keys (mamba_num_heads, ssm_state_size) are present.

DeepSeek-V4-Flashnot in its codenot stated
verdict and lines

There are no Mamba-2 SSD-style selective state-space blocks in the code; all layers are attention-based with compressors.

DeepSeek-V4-Pronot in its codenot stated
verdict and lines

No Mamba-2/SSD selective state-space block is implemented in this file.

MiMo-V2.5not in its codenot stated
verdict and lines

No Mamba-2 SSD selective state-space blocks are implemented; no mamba_num_heads or ssm_state_size config exists.

MiMo-V2.5-Pronot in its codenot stated
verdict and lines

No Mamba-2 SSD state-space block implementation or related config keys (mamba_num_heads, ssm_state_size) are present.

Qwen3.6-35B-A3Bnot in its codenot stated
verdict and lines

The linear-attention layers implement the gated delta rule (DeltaNet-style), not Mamba-2 SSD blocks; no mamba_num_heads or ssm_state_size config is present.

Gemma 4 31Bnot in its codenot stated
verdict and lines

No Mamba-2/SSD selective state-space block implementation is present in the modeling code.

Qwen3.5-397B-A17Bnot in its codenot stated
verdict and lines

The linear layers implement Gated DeltaNet, not Mamba-2 SSD blocks; there is no ssm_state_size or mamba_num_heads config, and no selective-SSM code.

DeepSeek-V3.2not in its codenot stated
verdict and lines

No Mamba-2/SSD selective state-space block implementation or related config keys (mamba_num_heads, ssm_state_size) exist in these files.

X-VLAnot in its codenot stated
verdict and lines

No Mamba-2 or SSM-style selective state-space block is implemented in the given modeling files.

gpt-oss-120bnot in its codenot stated
verdict and lines

No Mamba-2 SSM block implementation exists in this modeling file

OpenVLA / OpenVLA-OFTnot in its codenot stated
verdict and lines

No Mamba-2/SSD config keys (mamba_num_heads, ssm_state_size, hybrid_override_pattern) or code implementing selective state-space blocks are present.

In this branch 4

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 4

Mamba-2 core · 5 sources · 6 quotes
Mamba core · 2 sources · 2 quotes
2 single mentions
Mamba checkpointing core · 1 source · 1 quote
Mamba-2 SSM cache optional · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

↑