Which of this branch's techniques each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Model	filed here	token mixer	channel mixer	positional encoding	normalization & residual	prediction head	multimodal architecture	context capacity
GLM-5.3-Flash core	—	Hybrid Attention coreIndexPool coreKimi Delta Attention coreNoPE sparse multi-head latent attention core—	Mixture of Experts coreToken-level expert routing core—	—	Manifold-Constrained Hyper-Connections core—	—	Reusing the image token for video frames used—	—
DeepSeek-V4.1-Flash core	Asymmetric architecture coreCausal Encoder-Decoder (CED) architecture core—	Compressed Sparse Attention 2 coreCross-layer attention sharing coreCross-layer KV and index reuse with statically assigned CSA2 modes coreHierarchical Sparse Indexer coreReindex Mode coreReuse Mode coreSliding Window Attention coreCross-stage shared-state management for attention reuse usedCSA2 Full Mode usedDecoder SWA Bounded Replay usedFrom-scratch sparse attention training without dense warmup usedMicro-batch-level shared-state lifetime management usedProgressive sequence-length extension for sparse attention usedPure Sliding Window Attention usedSparse retrieval over long contexts evaluated—	Asymmetric input/output activation split coreAuxiliary-loss-free load balancing coreDeepSeekMoE shared and fine-grained routed experts coreMixture of Experts coreModality-specific auxiliary-loss-free load balancing coreMoE with 384 routed experts and a shared expert coreformHC usedMixture-of-Experts layers usedShared Experts usedSwiGLU usedSwiGLU clamping used—	2D rotary position embedding used—	Single-Pass mHC coreRMSNorm used—	—	3×3 pixel-unshuffle downsampling coreJoint vision-language pre-training coreMultimodal mixture-of-experts Transformer coreVision encoder–MLP projector pathway coreDisaggregated encoder training usedDiscarding the LLM after vision-encoder training usedHigh-resolution autoregressive vision-encoder fine-tuning usedLinear-projection patch embedding usedStaged vision-encoder freezing usedTwo-layer MLP vision projector used—	Engram coreRow-wise distributed partitioning of embedding tables used—
Hy4-preview core	—	Gated DeepSeek Sparse Attention coreIndexCache core—	Mixture of Experts coreMoE with routed and shared experts coreShared Experts coreTop-8 expert routing coreTop-8 Routed-Expert Selection with Shared-Expert Activation core—	—	Identity Hyper-Connections core—	Multi-Token Prediction core—	—	—
DeepSeek-V4-Flash-0731 core	—	—	—	—	—	—	Native multimodal visual understanding core—	—
NVIDIA-Nemotron-3-Ultra-550B-A55B core	—	Attention layers coreHybrid Mamba-Attention coreHybrid Mamba-Transformer coreMamba coreMamba-2 coreSparse Global Attention Anchors coreMamba-2 SSM cache optional—	Hybrid Latent Mixture of Experts coreLatentMoE coreMixture of Experts coreMixture-of-Experts layers coreExpert Parallelism Load Balancing usedShared Experts usedMaxVio evaluatedGranular Mixture of Experts not used—	—	—	Multi-Token Prediction coreSampled prior-MTP hidden-state conditioning usedShared-weight design across prediction heads usedNemotron Hybrid Multi-Token Prediction optional—	—	—
MiMo-V2.6-Flash core	—	Alternating Row-Major and Column-Major Token Serialization coreHybrid Attention coreHybrid sparse mixture-of-experts Transformer architecture coreSink-Augmented Sliding Window Attention core—	Mixture of Experts coreNo Shared Experts core—	—	—	Multi-Token Prediction core—	Four-frame audio patches with within-patch bidirectional self-attention coreJoint multimodal input and understanding coreTwo-stage audio encoding with tokenization and patch encoding coreData-parallel-first multimodal encoding usedJoint full-model multimodal training usedTwo-stage text-then-multimodal pre-training used—	1M-token context window core—
DeepSeek-V4-Flash core	—	Compressed Sparse Attention coreDeepSeek Sparse Attention coreHeavily Compressed Attention coreToken-wise compression coreAttention sink usedHybrid Attention usedMulti-query attention usedSliding Window Attention usedTwo-stage introduction of sparse attention used—	DeepSeekMoE coreMixture of Experts coreShared Experts coreAuxiliary-loss-free load balancing usedHash routing usedMegaMoE usedSwiGLU usedSwiGLU clamping usedAnticipatory Routing optionalLoss-spike-triggered Anticipatory Routing optional—	Rotary Position Embedding used—	Manifold-Constrained Hyper-Connections coreRMSNorm usedQK-Clip not used—	Multi-Token Prediction core—	—	1M-token context window default—
MiMo-V2.5 core	—	Global attention coreHybrid Attention coreHybrid Sliding Window Attention coreSliding Window Attention coreFull attention usedGrouped-query attention usedLearnable attention sink bias usedSWA-128 used—	Mixture of Experts coreDense feed-forward network usedExpert bias update factor usedLoad balancing usedMoE with 256 experts and top-8 routing usedTop-8 expert routing usedRound-robin load balancing optionalNo Shared Experts not used—	Rotary Position Embedding used—	—	Multi-Token Prediction core—	Native multimodal understanding coreNative visual and audio understanding coreAudio encoder initialized from MiMo-Audio usedProjector warmup usedVision Transformer (ViT) usedVisual and audio encoders with lightweight projectors used—	1M-token context window usedProgressive context extension used—
Hy3 core	—	Grouped-query attention core—	Mixture of Experts coreRouted experts coreShared Experts coreTop-8 expert routing core—	—	—	Multi-Token Prediction core—	—	—
GLM-5.2 core	—	DeepSeek Sparse Attention coreIndexShare coreMulti-Head Latent Attention coreLightning Indexer usedSparse attention usedGated DeltaNet evaluatedSearch-Based Sliding-Window Attention Pattern evaluatedSimpleGDN evaluatedSliding Window Attention evaluatedMulti-query attention mentioned—	Mixture of Experts coreDP-aware routing used—	—	—	Multi-Token Prediction used—	—	—
MiniMax-M3 core	—	Block-level selection coreBlock-sparse grouped-query attention coreDynamic sparse selection coreGrouped-query attention coreIndex Branch coreMain Branch coreMiniMax Sparse Attention coreSequential block processing coreSparse attention coreBlock Max Pooling usedIndexer Warmup usedKV-outer sparse attention usedLocal Block usedNatively trained sparsity usedSingle-head index key usedSparse softmax attention usedTop-k block selection usedTwo-stage sparse attention usedDeepSeek Sparse Attention evaluatedFixed Sliding Window Attention evaluatedFull attention evaluatedMixture of Block Attention evaluatedLinear attention mentionedSliding Window Attention mentionedCompressed Sparse Attention not usedDense Attention Fallback unclearIndex Branch output not usedIndex Branch value head not usedLearnable attention sink bias not usedLightning Attention not usedMulti-Head Latent Attention not usedNative Sparse Attention not usedRoPE with attention sink unclear—	Mixture of Experts corePersistent load balancing usedShared Experts usedTop-4 expert routing used—	Rotary Position Embedding used—	—	—	Mixed-modality training core—	—
DeepSeek-V3.2 core	—	DeepSeek Sparse Attention coreFine-grained token selection coreLightning Indexer coreMQA Mode of Multi-Head Latent Attention coreDense Warm-up Stage usedDetached indexer-input optimization usedMulti-Head Latent Attention MHA/MQA Modes usedSparse-attention continued pre-training with joint model and indexer optimization used—	Keep Routing core—	—	—	—	—	—
DeepSeek-V4-Flash-Vision-Exp core	—	DFlash attention core—	Mixture of Experts core—	—	Hyper-Connections core—	—	Native multimodal visual understanding coreVision encoder and multimodal aligner coreVisual modules for multimodal understanding core—	—
DeepSeek-V4-Pro core	—	Compressed Sparse Attention coreDeepSeek Sparse Attention coreHeavily Compressed Attention coreToken-wise compression coreAttention sink usedHybrid Attention usedMulti-query attention usedSliding Window Attention usedTwo-stage introduction of sparse attention used—	DeepSeekMoE coreMixture of Experts coreShared Experts coreAnticipatory Routing usedAuxiliary-loss-free load balancing usedHash routing usedMegaMoE usedSwiGLU usedSwiGLU clamping usedLoss-spike-triggered Anticipatory Routing optional—	Rotary Position Embedding used—	Manifold-Constrained Hyper-Connections coreRMSNorm usedQK-Clip not used—	Multi-Token Prediction core—	—	1M-token context window defaultEngram not used—
DeepSeek-V4-Pro-0813 core	—	—	—	—	—	—	Native multimodal visual understanding core—	—
Gemma 4 31B core	Encoder-free decoder-only Transformer architecture corePer-Layer Embeddings (PLE) coreDense model architecture used—	Hybrid Attention coreUnified Keys and Values core—	Mixture of Experts coreSparse expert activation core—	2D coordinate-based positional embeddings used2D rotary position embedding usedProportional Rotary Position Embedding used—	—	—	Encoder-free multimodal architecture coreDedicated multimodal encoders usedFrozen encoders during pre-training usedLightweight vision embedding module usedRaw audio projection into the LLM embedding space usedSingle-matrix-multiplication vision projection usedUniversal Speech Model (USM)-based audio encoder usedVariable aspect-ratio image handling usedConfigurable visual token budget optional—	—
Inkling core	Regular Transformer decoder coreShort convolutions in attention and residual branches used—	Hybrid Attention core512-token Sliding Window Attention usedGrouped-query attention usedKernel-4 convolutional layers in decoder blocks used—	Mixture of Experts coreMoE with Expert Sinks coreShared Experts Active on Every Token coreToken-level expert routing coreTop-6 expert routing coreSigmoid-based MoE router with auxiliary-loss-free load balancing used—	Relative attention coreLearned input-dependent relative position bias used—	Post-Embedding RMSNorm used—	—	Discrete token encoding for audio coreHierarchical patch encoder coreJoint multimodal decoding in a shared hidden space coreNative multimodal encoding coreEncoder-free multimodal architecture used—	—
Kimi K3 core	—	Gated Multi-Head Latent Attention coreHybrid Attention coreHybrid linear attention coreKDA coreKimi Delta Attention coreKimi Delta Attention and Attention Residuals architecture coreFactorized spatial-temporal video attention with temporal pooling usedGated DeltaNet usedKimi Delta Attention with input-dependent full-rank output gate usedKimi Delta Attention with lower-bounded log-decay used—	Auxiliary-loss-free load balancing coreDispatch recomputation coreLatentMoE coreMixture of Experts coreNormalized LatentMoE coreOutput-independent MoE gradient reformulation coreQuantile Balancing coreSharded latent weights with fused all-gather GEMM epilogue coreSiTU-GLU coreSparse expert activation coreFixed Top-k routing with frozen bias defaultExact coordinate minimization usedExponential moving average of estimated quantiles usedHistogram-based quantile estimation usedLatent-space routing usedPooled-global-batch quantile estimation usedSiTU used—	No Position Encoding core—	Attention Residuals (AttnRes) coreBlock AttnRes coreTwo-Phase Block AttnRes Schedule coreFull Attention Residuals mentioned—	—	Single shared multimodal backbone core2×2 pixel-shuffle downsampling usedDecoupled Encoder Process (DEP) usedDual-format coordinate supervision usedFrom-scratch vision-encoder training with next-token prediction usedMoonViT-V2 usedMultimodal input used—	Progressive context extension used—
Laguna-S-2.1 core	—	Grouped-query attention coreHybrid Attention coreSoftplus attention gating coreGated Attention and Sliding-Window Attention Configuration usedSoftplus-based per-head gating used512-token Sliding Window Attention evaluatedDense gated attention with full RoPE and full gating evaluatedPer-Head Gating evaluatedSWA-1024 evaluated—	Mixture of Experts coreSparse expert activation coreToken-choice routing with softplus gating coreAuxiliary-loss load balancing usedRouted expert output modulation usedShared Experts usedToken-choice routing used—	Per-layer-type rotary position scales coreRotary Position Embedding usedGated attention with partial RoPE evaluated—	—	—	—	—
MiMo-V2.5-Pro core	—	Hybrid Attention coreGlobal attention usedHybrid Sliding Window Attention usedLearnable attention sink bias usedSliding Window Attention used—	Mixture of Experts core—	—	—	Multi-Token Prediction used—	Native multimodal understanding coreNative visual and audio understanding coreProjector warmup usedVisual and audio encoders with lightweight projectors used—	1M-token context window usedProgressive context extension used—
MiMo-V2.6-Pro core	—	Alternating Row-Major and Column-Major Token Serialization coreHybrid Attention coreHybrid sparse mixture-of-experts Transformer architecture coreSink-Augmented Sliding Window Attention core—	Mixture of Experts coreNo Shared Experts coreSparse expert activation core—	—	—	Multi-Token Prediction core—	Four-frame audio patches with within-patch bidirectional self-attention coreTwo-stage audio encoding with tokenization and patch encoding coreData-parallel-first multimodal encoding usedJoint full-model multimodal training usedTwo-stage text-then-multimodal pre-training used—	—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B core	—	Hybrid Mamba-Transformer coreMamba-2 coreMamba-2, Mixture-of-Experts, and Selective Attention Hybrid core—	LatentMoE coreMixture of Experts coreToken-level expert routing coreLatent-dimension reduction usedMoE with 128 routed experts and a shared expert used—	Omitting RoPE in attention layers core—	—	Multi-Token Prediction core—	—	—
Qwen3.5-397B-A17B core	—	DeepSeek Sparse Attention coreGated DeltaNet coreGated DeltaNet and Full Attention coreGated DeltaNet with reduced KV-head configuration coreGated DeltaNet–sparse MoE hybrid corehybrid Gated DeltaNet + sparse MoE architecture coreHybrid Mamba-Transformer Mixture-of-Experts Layer Layout coreLinear attention coreDynamic Attention Window Size Training usedMulti-Head Attention (MHA) used—	10 Routed + 1 Shared Experts Activated per Token core8 Routed + 1 Shared Experts Activated per Token coreHybrid Mixture of Experts coreMixture of Experts coreSparse expert activation core—	YaRN usedRoPE scaling optionalTemporal-Modality Rotary Position Embedding not used—	—	Multi-Token Prediction usedMulti-Token Prediction for residual codebooks used—	Early fusion multimodal training coreThinker–Talker architecture coreAudio Transformer (AuT) usedCausal streaming ConvNet codec decoder usedExplicit text-string timestamps usedMel-spectrogram audio front end usedResidual Vector Quantization (RVQ) speech representation usedVideo preprocessor longest-edge configuration optional—	—
Qwen3.6-35B-A3B core	—	Gated DeltaNet coreGated DeltaNet and Gated Attention core—	Gated DeltaNet MoE coreMixture of Experts core—	RoPE scaling optionalYaRN optional—	—	—	Early fusion multimodal training core—	—
Qwen3.8-Flash-Next core	—	Compressed lightweight indexer coreFixed-budget sparse-attention selection coreGated DeltaNet coreGated DeltaNet and Qwen Sparse Attention coreHybrid Attention coreMQA indexer coreQSA micro-block compression at ratio 4 coreQwen Sparse Attention coreReuse QSA index selection across speculative decoding steps coreTop-512 block selection coreAverage pooling usedBlock-causal scoring usedGated DeltaNet with bounded sigmoid output gate usedJoint backbone and indexer training under sparse attention usedTop-scoring compressed-block selection used—	10 Routed + 1 Shared Experts Activated per Token coreMixture of Experts coreContextual gating for N-gram embedding injection usedFrequency-based partitioning of N-gram embedding slots evaluatedSwiGLU clipping not used—	Partial RoPE usedYaRN used—	All-branch Elementwise Read Gate coreDynamic Gating of Residual Reads and Writes coreElementwise Data-Dependent Residual Read Gate coreFour-Branch Residual Stream coreFour-Stream Hyper-Connection Combine Update coreGated Residual coreGatedNorm coreIndependent Per-Branch Normalization coreLow-Rank Gated Mix corePer-Branch Scalar Write Gate coreWidened Residual Stream coreBounded Positive Gates usedData-Dependent Residual Read and Write Operators usedZero-Centered RMSNorm usedHyper-Connections evaluatedManifold-Constrained Hyper-Connections evaluatedSimplified AltUp evaluatedQK-Clip not usedSparse Gated Residual Writes not used—	Multi-Token Prediction core—	—	N-gram embedding host-memory offload and prefetch coreN-gram embedding lookup coreSingle-layer N-gram embedding placement coreMulti-head hashing usedToken normalization for N-gram vocabulary compression evaluated—
Step-3.7-Flash core	—	Cascade attention not used—	Mixture of Experts core—	—	—	Multi-Token Prediction optional—	Language backbone coreVision encoder coreVision Transformer (ViT) used—	256K context window core—
gpt-oss-120b core	—	Grouped-query attention usedHybrid Attention usedLearned softmax denominator bias used—	Mixture of Experts coreTop-k expert routing with softmax over selected experts coreSwiGLU used—	Rotary Position Embedding usedYaRN used—	Pre-LN usedRMSNorm used—	—	—	—