methods · linear attention & state space
Mamba
4 methods filed at this node or below it, from the sources of 3 models.
In practice
Checked as “Mamba-2 layers” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | in its code | not stated | mamba_num_heads=128 · ssm_state_size=128 · mamba_head_dim=64 · chunk_size=128The text_config declares Mamba-2-style SSM hyperparameters (ssm_state_size, mamba_num_heads, mamba_head_dim, chunk_size) used by the 'mamba' layers in layers_block_type, matching NemotronH's Mamba-2 mixer design. llm_config.ssm_state_size = 128llm_config.mamba_num_heads = 128llm_config.mamba_head_dim = 64llm_config.chunk_size = 128 | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | in its code | adopted | mamba_num_heads=64 · ssm_state_size=128The 'mamba' layers use NemotronHMamba2Mixer implementing the Mamba-2 SSD chunked scan with per-head scalar decay (A_log, D, dt_bias) and ssm_state_size. mamba_num_heads = 64ssm_state_size = 128 | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | in its code | adopted | mamba_num_heads=256 · ssm_state_size=128The linear-attention layers are implemented as NemotronHMamba2Mixer, a Mamba-2 selective state-space block with chunked SSD scan and per-head scalar decay (A_log/D), configured via mamba_num_heads and ssm_state_size. mamba_num_heads = 256ssm_state_size = 128 | code → |
| Mistral-Large-4 | not in its code | not stated | verdict and linesNo Mamba-2/SSD state-space block implementation or related config keys (mamba_num_heads, ssm_state_size) are present. | code → |
| LFM3-1.7B | not in its code | not stated | verdict and linesNo Mamba-2 SSD selective state-space block is implemented; non-attention layers are short causal convolutions (Lfm3ShortConv), not SSM. | code → |
| MiniMax-M3.1-Flash | not in its code | not stated | verdict and linesNo Mamba-2/SSD state-space block exists in the given modeling code. | code → |
| DeepSeek-V4.1-Flash-1017 | not in its code | not stated | verdict and linesNo Mamba-2 SSD state-space blocks are implemented or referenced in the config or code. | code → |
| Step-5-Preview | not in its code | not stated | verdict and linesNo Mamba-2 SSD blocks are implemented or referenced; the model is a GQA transformer with sparse attention. | code → |
| Kolibri-1 | not in its code | not stated | verdict and linesNo SSM/Mamba-2 state-space block or related config keys (mamba_num_heads, ssm_state_size) appear in this model. | code → |
| MiMo-V2.6-Flash | not in its code | not stated | verdict and linesNo Mamba-2/SSD selective state-space block is implemented anywhere in the given files. | code → |
| MiMo-V2.6-Pro | not in its code | not stated | verdict and linesNo Mamba-2/SSD selective state-space block is implemented anywhere in the modeling file. | code → |
| DeepSeek-V4.1-Flash | not in its code | not stated | verdict and linesNo Mamba-2 SSD state-space block implementation is present in the provided code or config. | code → |
| Hy4-preview | not in its code | not stated | verdict and linesNo Mamba-2/SSD state-space block implementation or related config keys (mamba_num_heads, ssm_state_size) are present. | code → |
| GLM-5.3-Flash | not in its code | not stated | verdict and linesNo Mamba-2/SSD mechanism (no mamba_num_heads, ssm_state_size, or per-head scalar decay SSM module) exists; the recurrent mixer implemented is Kimi Delta Attention, not Mamba-2. | code → |
| Qwen3.8-Flash-Next | not in its code | not stated | verdict and linesNo Mamba-2/SSD selective state-space block is implemented; the recurrent mixer present is the gated delta rule (GatedDeltaNet), not Mamba-2. | code → |
| DeepSeek-V4-Flash-Vision-Exp | not in its code | not stated | verdict and linesNo Mamba-2 selective state-space block is implemented in this file; there is no ssm_state_size or mamba head config and no SSD code path. | code → |
| GLM-5.3 | not in its code | not stated | verdict and linesNo Mamba-2/SSD selective state-space block code exists in the given files. | code → |
| DeepSeek-V4-Pro-0813 | not in its code | not stated | verdict and linesNo Mamba-2 SSD selective state-space block is implemented in the modeling file | code → |
| DeepSeek-V4-Flash-0731 | not in its code | not stated | verdict and linesNo Mamba-2 selective state-space blocks are implemented in the modeling file. | code → |
| Xiaomi-Robotics-1 | not in its code | not stated | verdict and linesNo Mamba-2 SSD blocks or ssm_state_size/mamba_num_heads config exist; the model is a standard transformer decoder. | code → |
| Laguna-S-2.1 | not in its code | not stated | verdict and linesNo Mamba-2 SSM block or mamba_num_heads/ssm_state_size config exists; all layers are LagunaAttention. | code → |
| Kimi K3 | not in its code | not stated | verdict and linesNo Mamba-2/SSD selective state-space block (mamba_num_heads, ssm_state_size) exists in the code or config; the linear-attention layers use KDA (delta-rule), not Mamba-2. | code → |
| Inkling | not in its code | not stated | verdict and linesThere are no Mamba-2 SSD blocks; config only carries short-conv (sconv) parameters, not SSM state sizes or per-head decay. | code → |
| Hy3 | not in its code | not stated | verdict and linesNo SSM/Mamba-2 selective state-space block or related config keys (mamba_num_heads, ssm_state_size) appear in this model. | code → |
| GLM-5.2 | not in its code | not stated | verdict and linesNo Mamba-2 SSD state-space blocks are implemented in the provided modeling file. | code → |
| MiniMax-M3 | not in its code | not stated | verdict and linesNo Mamba-2 SSM blocks or related config keys (mamba_num_heads, ssm_state_size) are present. | code → |
| Step-3.7-Flash | not in its code | not stated | verdict and linesNo Mamba-2 SSD block or mamba-related config keys (mamba_num_heads, ssm_state_size) are present in the code or config. | code → |
| MolmoAct 2 | not in its code | not stated | verdict and linesNo Mamba-2/SSD selective state-space block implementation or related config keys (mamba_num_heads, ssm_state_size) are present. | code → |
| DeepSeek-V4-Flash | not in its code | not stated | verdict and linesThere are no Mamba-2 SSD-style selective state-space blocks in the code; all layers are attention-based with compressors. | code → |
| DeepSeek-V4-Pro | not in its code | not stated | verdict and linesNo Mamba-2/SSD selective state-space block is implemented in this file. | code → |
| MiMo-V2.5 | not in its code | not stated | verdict and linesNo Mamba-2 SSD selective state-space blocks are implemented; no mamba_num_heads or ssm_state_size config exists. | code → |
| MiMo-V2.5-Pro | not in its code | not stated | verdict and linesNo Mamba-2 SSD state-space block implementation or related config keys (mamba_num_heads, ssm_state_size) are present. | code → |
| Qwen3.6-35B-A3B | not in its code | not stated | verdict and linesThe linear-attention layers implement the gated delta rule (DeltaNet-style), not Mamba-2 SSD blocks; no mamba_num_heads or ssm_state_size config is present. | code → |
| Gemma 4 31B | not in its code | not stated | verdict and linesNo Mamba-2/SSD selective state-space block implementation is present in the modeling code. | code → |
| Qwen3.5-397B-A17B | not in its code | not stated | verdict and linesThe linear layers implement Gated DeltaNet, not Mamba-2 SSD blocks; there is no ssm_state_size or mamba_num_heads config, and no selective-SSM code. | code → |
| DeepSeek-V3.2 | not in its code | not stated | verdict and linesNo Mamba-2/SSD selective state-space block implementation or related config keys (mamba_num_heads, ssm_state_size) exist in these files. | code → |
| X-VLA | not in its code | not stated | verdict and linesNo Mamba-2 or SSM-style selective state-space block is implemented in the given modeling files. | code → |
| gpt-oss-120b | not in its code | not stated | verdict and linesNo Mamba-2 SSM block implementation exists in this modeling file | code → |
| OpenVLA / OpenVLA-OFT | not in its code | not stated | verdict and linesNo Mamba-2/SSD config keys (mamba_num_heads, ssm_state_size, hybrid_override_pattern) or code implementing selective state-space blocks are present. | code → |
In this branch 4
Everything filed at this node or below it, with one collapsible heading per child node.
filed here 4
By model
Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.
| Model | methods |
|---|---|
| NVIDIA-Nemotron-3-Ultra-550B-A55B core | Mamba coreMamba-2 coreMamba-2 SSM cache optional— |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B core | Mamba-2 core— |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B core | Mamba checkpointing core— |