methods · token mixer
hybrid layer stacking
12 methods filed at this node or below it, from the sources of 22 models.
In practice
Checked as “Interleaved sliding-window and global attention” against each model’s own code and config.json, apart from its documents.
| Model | Code | Its documents | Parameters | Link |
|---|---|---|---|---|
| LFM3-1.7B | in its code | not stated | layer_types=["conv", "conv", "full_attention", …layer_types interleaves full_attention and sliding_attention layers (plus conv layers) in a fixed pattern, with separate mask builders per type. layer_types = ["conv", "conv", "full_attention", "conv", "sliding_attention", "conv", "full… | code → |
| DeepSeek-V4.1-Flash-1017 | in its code | not stated | compress_ratios=[0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2…compress_ratios interleaves layers with compress_ratio 0 (SWA-only, no compressed/global KV) against layers with ratio 2 or 1 (which read a compressed, effectively global, KV cache), giving a fixed interleaving of local and global-context attention. text_config.compress_ratios = [0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1… | code → |
| Step-5-Preview | in its code | not stated | layer_types=["sliding_attention", "sliding_atte…layer_types fixes a repeating pattern of three sliding_attention layers followed by one full_attention layer across the 92-layer stack. text_config.layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "full_attenti… | code → |
| Kolibri-1 | in its code | adopted | layer_types=["sliding_attention", "sliding_atte…layer_types interleaves four sliding_attention layers followed by one full_attention layer, repeating across all 50 layers. layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "sliding_atte… | code → |
| MiMo-V2.6-Flash | in its code | adopted | hybrid_layer_pattern=[0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0…hybrid_layer_pattern interleaves full-attention layers (0) with sliding-window layers (1) in a fixed repeating pattern across the 48 decoder layers. hybrid_layer_pattern = [0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1… | code → |
| MiMo-V2.6-Pro | in its code | adopted | hybrid_layer_pattern=[0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1…hybrid_layer_pattern interleaves full-attention layers (0) with sliding-window layers (1) in a fixed repeating pattern across the 70 layers. hybrid_layer_pattern = [0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1… | code → |
| DeepSeek-V4.1-Flash | in its code | not stated | compress_ratios=[0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2… · kv_source_layer_ids=[2, 8, 14, 20]compress_ratios defines a per-layer pattern where some layers are SWA-only (ratio 0) and others combine SWA with a compressed/global KV source (ratio 1 or 2), giving a fixed interleaving of local and global attention across the 40 layers. text_config.compress_ratios = [0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1…text_config.kv_source_layer_ids = [2, 8, 14, 20] | code → |
| DeepSeek-V4-Flash-Vision-Exp | in its code | not stated | verdict and lineslayer_types interleaves sliding_attention, compressed_sparse_attention, and heavily_compressed_attention layers, each with different full/compressed context handling, per the compress_ratios-derived layer_types. | code → |
| Laguna-S-2.1 | in its code | adopted | layer_types=["full_attention", "sliding_attenti…layer_types fixes a pattern of one full_attention layer followed by three sliding_attention layers, repeated across all 48 layers. layer_types = ["full_attention", "sliding_attention", "sliding_attention", "sliding_attenti… | code → |
| Inkling | in its code | adopted | local_layer_ids=[0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 12,… · num_hidden_layers=66local_layer_ids selects most layers as sliding-window (swa_*) while layers not listed (e.g. 5, 11, 17...) use full attention, giving a fixed interleaving of local and global attention layers. text_config.local_layer_ids = [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 24, 2…text_config.num_hidden_layers = 66 | code → |
| Step-3.7-Flash | in its code | not stated | layer_types=["full_attention", "sliding_attenti…layer_types interleaves one full_attention layer every 4 layers with three sliding_attention layers, and the model builds separate causal_mask_mapping entries for each. text_config.layer_types = ["full_attention", "sliding_attention", "sliding_attention", "sliding_attenti… | code → |
| DeepSeek-V4-Flash | in its code | not stated | compress_ratios=[0, 0, 4, 128, 4, 128, 4, 128, 4, 1…layer_types interleaves sliding_attention (plain local window) layers with compressed_sparse_attention and heavily_compressed_attention layers that attend over long-range compressed history in addition to the local window, giving a fixed interleaving of local vs. extended-context attention per the compress_ratios config. compress_ratios = [0, 0, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128… | code → |
| MiMo-V2.5 | in its code | adopted | hybrid_layer_pattern=[0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0…hybrid_layer_pattern interleaves full-attention layers (0) every 6 layers with sliding-window layers (1) for the rest, and the decoder layer sets attention_type accordingly. hybrid_layer_pattern = [0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1… | code → |
| MiMo-V2.5-Pro | in its code | adopted | verdict and lineshybrid_layer_pattern interleaves full-attention layers (0) with sliding-window layers (1) in a fixed repeating pattern across 70 layers. | code → |
| Gemma 4 31B | in its code | adopted | layer_types=["sliding_attention", "sliding_atte…layer_types interleaves 5 sliding_attention layers followed by 1 full_attention layer repeatedly across the 60 layers. text_config.layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "sliding_atte… | code → |
| gpt-oss-120b | in its code | adopted | layer_types=["sliding_attention", "full_attenti…layer_types alternates sliding_attention and full_attention across the 36 layers, each built with its own causal mask layer_types = ["sliding_attention", "full_attention", "sliding_attention", "full_attention"… | code → |
| Mistral-Large-4 | not in its code | not stated | verdict and lineslayer_types lists only 'indexed_attention' for every layer, with no sliding-window variant interleaved. | code → |
| MiniMax-M3.1-Flash | not in its code | not stated | verdict and lineslayer_types only distinguishes full_attention from the sparse-indexer attention type; there is no sliding-window layer variant interleaved with global attention. | code → |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B | not in its code | not stated | sliding_window=nullSince sliding_window is null, there is no interleaving of sliding and full attention; layer typing only alternates mamba/moe/attention, not windowed vs global attention. llm_config.sliding_window = null | code → |
| Hy4-preview | not in its code | not stated | layer_types=["deepseek_sparse_attention", "deep…There is no interleaving of sliding-window and global layers; every layer_types entry is the same 'deepseek_sparse_attention' type. layer_types = ["deepseek_sparse_attention", "deepseek_sparse_attention", "deepseek_sparse_a… | code → |
| GLM-5.3-Flash | not in its code | not stated | layer_types=["linear_attention", "linear_attent…There is no sliding-window layer type at all (layer_types only contains linear_attention and deepseek_sparse_attention/indexed_attention), so there is no interleaving of sliding and global attention. text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "deepseek_sparse… | code → |
| Qwen3.8-Flash-Next | not in its code | not stated | layer_types=["linear_attention", "linear_attent…layer_types interleaves linear_attention with full/indexed attention, not sliding-window with global attention, so there is no sliding-window/global interleaving. text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"… | code → |
| GLM-5.3 | not in its code | not stated | verdict and lineslayer_types is uniformly "indexed_attention" for all layers with no sliding/global interleaving pattern present. | code → |
| DeepSeek-V4-Pro-0813 | not in its code | not stated | sliding_window=128All layers apply the same sliding-window K cache regardless of layer_types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention all share the sliding-window branch); the compressor variants add extra long-range compressed KV on top, but none attend to the literal full context without a window, so there is no interleaving of sliding vs full/global attention sliding_window = 128 | code → |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B | not in its code | not stated | sliding_window=nulllayers_block_type only distinguishes mamba/moe/attention (full_attention), with no sliding-window layer type or pattern present. sliding_window = null | code → |
| DeepSeek-V4-Flash-0731 | not in its code | not stated | sliding_window=128All layers use the same sliding-window base attention (sliding_window=128 applies uniformly); the layer_types distinguish only whether a compressor (CSA/HCA) is added on top, not whether full global attention replaces sliding-window, so there is no interleaving of sliding vs full-context layers. sliding_window = 128 | code → |
| Xiaomi-Robotics-1 | not in its code | not stated | verdict and linesThere is no layer_types or sliding_window_pattern config, and no code that interleaves local and global attention layers. | code → |
| Kimi K3 | not in its code | not stated | verdict and linesThere is no interleaving of sliding-window and global attention layers; the fixed interleaving present is between full-attention (MLA) and linear-attention (KDA) layers, not local-window vs global attention. | code → |
| Hy3 | not in its code | not stated | verdict and linesThere is no layer_types or sliding_window_pattern config, and all attention layers use the same full causal mask. | code → |
| GLM-5.2 | not in its code | not stated | verdict and linesThere is no layer_types/sliding_window_pattern distinguishing local vs global attention; all 78 layers share the same "indexed_attention" type. | code → |
| NVIDIA-Nemotron-3-Ultra-550B-A55B | not in its code | not stated | sliding_window=nullThere is no sliding-window layer type in layers_block_type (only mamba, moe, attention), so no interleaving of local/global attention occurs. sliding_window = null | code → |
| MiniMax-M3 | not in its code | not stated | verdict and lineslayer_types alternates only between full_attention and the lightning-indexer sparse attention, not sliding-window vs global attention. | code → |
| MolmoAct 2 | not in its code | not stated | verdict and linesNo layer_types or sliding_window_pattern exist in config and all decoder layers use the same MolmoAct2Attention class uniformly, so there is no interleaving of sliding and full attention. | code → |
| DeepSeek-V4-Pro | code unclear | not stated | sliding_window=128layer_types is derived at runtime from compress_ratios (ending with a 0 which maps to sliding_attention, others to compressed_sparse_attention/heavily_compressed_attention), but all layer types still apply the same sliding_window base cache, and compress-attention layers additionally concatenate long-range compressed KV rather than attending to the genuinely full context, so whether any layer is truly 'full-context' versus all being sliding-window-based cannot be settled from the given files. sliding_window = 128 | code → |
| Qwen3.6-35B-A3B | not in its code | not stated | verdict and lineslayer_types interleaves linear_attention and full_attention, not sliding-window and global attention; no sliding-window mechanism exists. | code → |
| Qwen3.5-397B-A17B | not in its code | not stated | layer_types=["linear_attention", "linear_attent…layer_types interleaves full_attention and linear_attention (not sliding-window) layers, so this is a linear/full hybrid, not sliding-window/full. text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"… | code → |
| DeepSeek-V3.2 | not in its code | not stated | verdict and linesThere is no layer_types or sliding_window_pattern distinguishing local vs global attention; every layer is uniformly 'indexed_attention'. | code → |
| X-VLA | not in its code | not stated | verdict and linesThere is no interleaving of sliding and full-attention layers; all layers use the same uniform full/causal attention pattern. | code → |
| OpenVLA / OpenVLA-OFT | not in its code | not stated | model_type=llamaNo layer_types or sliding_window_pattern configuration exists, and no interleaving code is present; attention layers are uniform. text_config.model_type = "llama" | code → |
In this branch 12
Everything filed at this node or below it, with one collapsible heading per child node.
filed here 12
By model
Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.