| DeepSeek-V4.1-Flash-1017 core | Causal Encoder-Decoder (CED) core— | — | Asymmetric input/output parameter activation coreMixture of experts core— | — | — | — | Joint vision-language pre-training from the start core— | — |
| GLM-5.3-Flash core | — | Hybrid Sparse and Linear Attention coreIndexPool coreKimi Delta Attention coreNoPE sparse multi-head latent attention core— | Mixture of experts coreTop-k expert routing core— | — | Manifold-Constrained Hyper-Connections core— | — | Reusing image tokens for video frames used— | — |
| DeepSeek-V4.1-Flash core | Asymmetric architecture coreCausal Encoder-Decoder (CED) core— | Compressed Sparse Attention 2 coreCSA2 compressor and indexer design coreCSA2 Full, Reindex, and Reuse modes coreHierarchical Sparse Indexer coreDecoder SWA Bounded Replay usedProgressive context-length extension for sparse attention usedSparse attention training from scratch without dense-attention warmup used— | Asymmetric input/output parameter activation coreMixture of experts coreModality-specific load balancing core— | 2D rotary position embeddings used— | Single-Pass mHC coreManifold-Constrained Hyper-Connections used— | — | Joint vision-language pre-training from the start core3×3 PixelUnshuffle downsampling usedTwo-layer MLP vision projector used— | Engram core— |
| Hy4-preview core | — | Gated DeepSeek Sparse Attention coreIndexCache core— | Mixture of experts coreShared expert coreTop-k expert routing core— | — | Identity Hyper-Connections core— | MTP speculative decoding core— | — | — |
| Step-5-Preview core | — | — | Mixture of experts core— | — | — | — | Multimodal input understanding core— | 1M-token context window coreEnabling 1M context in Step Plan optional— |
| MiniMax-M3.1-Flash core | — | MiniMax Sparse Attention coreSparse attention coreAggressive sparsity for inference efficiency used— | Mixture of experts core— | — | — | — | — | — |
| DeepSeek-V4-Flash-0731 core | — | — | — | — | — | — | Native multimodal architecture core— | — |
| Hy3 core | — | Grouped-query attention core— | Mixture of experts coreShared expert coreSparse expert activation coreTop-k expert routing core— | — | — | Multi-Token Prediction core— | — | — |
| MiMo-V2.6-Flash core | — | Alternating Row-Major and Column-Major Token Serialization coreHybrid Sliding-Window and Global Attention coreHybrid sparse mixture-of-experts Transformer architecture coreSink-Augmented Sliding Window Attention core— | Mixture of experts coreMoE without shared experts core— | — | — | Multi-Token Prediction core— | Four-frame audio patches with within-patch bidirectional attention coreJoint multimodal input understanding coreTwo-stage audio encoding with tokenization and patch encoding coreData-parallel-first multimodal encoding usedJoint full-model multimodal training usedTwo-stage text-then-multimodal pre-training used— | 1M-token context window core— |
| NVIDIA-Nemotron-3-Ultra-550B-A55B core | — | Attention layers coreHybrid Mamba-Attention coreHybrid Mamba-Transformer coreMamba coreMamba-2 coreSparse Global Attention Anchors coreMamba-2 SSM cache optional— | Hybrid LatentMoE coreLatentMoE coreMixture of experts coreExpert Parallelism Load Balancing usedShared expert usedMaxVio evaluatedExpert granularity not used— | — | — | MTP speculative decoding coreMulti-Token Prediction coreHidden-state mixing during recursive MTP training usedShared Multi-Token Prediction head usedNemotron Hybrid Multi-Token Prediction optional— | — | — |
| DeepSeek-V4-Flash core | — | Compressed Sparse Attention coreDeepSeek Sparse Attention coreHeavily Compressed Attention coreToken-wise compression coreAttention sink usedHybrid compressed attention (CSA and HCA) usedMulti-query attention usedSliding Window Attention usedTwo-stage sparse-attention training used— | DeepSeekMoE coreMixture of experts coreShared expert coreAuxiliary-loss-free load balancing usedHash routing usedMegaMoE usedSwiGLU usedSwiGLU clamping usedAnticipatory Routing optionalLoss-spike-triggered Anticipatory Routing optional— | Rotary Position Embedding used— | Manifold-Constrained Hyper-Connections coreRMSNorm usedQK-Clip not used— | Multi-Token Prediction core— | — | 1M-token context window default— |
| Alpamayo 1.5 / 2 Super core | — | — | — | — | — | Diffusion-based trajectory decoder core— | Chain-of-thought reasoning vision-language-action model coreSeparate flow-matching action expert coreEnd-to-end autonomous driving usedTriplane-based multi-camera tokenization evaluated— | — |
| DeepSeek-V3.2 core | — | DeepSeek Sparse Attention coreFine-grained token selection mechanism coreLightning Indexer coreMQA mode of MLA coreDense attention warm-up with a frozen lightning indexer usedDetached indexer-input optimization usedMHA and MQA modes of MLA usedSparse-attention continued pretraining with joint model and indexer optimization used— | Keep Routing core— | — | — | — | — | — |
| DeepSeek-V4-Flash-Vision-Exp core | — | DFlash attention core— | Mixture of experts core— | — | Hyper-Connections core— | — | Native multimodal architecture coreVision encoder and multimodal aligner coreVisual modules for multimodal understanding core— | — |
| DeepSeek-V4-Pro core | — | Compressed Sparse Attention coreDeepSeek Sparse Attention coreHeavily Compressed Attention coreToken-wise compression coreAttention sink usedHybrid compressed attention (CSA and HCA) usedMulti-query attention usedSliding Window Attention usedTwo-stage sparse-attention training used— | DeepSeekMoE coreMixture of experts coreShared expert coreAnticipatory Routing usedAuxiliary-loss-free load balancing usedHash routing usedMegaMoE usedSwiGLU usedSwiGLU clamping usedLoss-spike-triggered Anticipatory Routing optional— | Rotary Position Embedding used— | Manifold-Constrained Hyper-Connections coreRMSNorm usedQK-Clip not used— | Multi-Token Prediction core— | — | 1M-token context window defaultEngram not used— |
| DeepSeek-V4-Pro-0813 core | — | — | — | — | — | — | Native multimodal architecture core— | — |
| FLUX 3 Action core | Diffusion Transformer (DiT) core— | — | — | — | — | — | Embodiment-specific action heads coreRobot-specific input and output layers coreShared action trunk coreShared frozen encoders used— | — |
| GLM-5.2 core | — | DeepSeek Sparse Attention coreIndexShare coreMulti-head Latent Attention coreLightning Indexer usedSparse attention usedGated DeltaNet evaluatedSearch-Based Sliding-Window Attention Pattern evaluatedSimpleGDN evaluatedSliding Window Attention evaluatedMulti-query attention mentioned— | Mixture of experts coreDP-aware routing used— | — | — | MTP speculative decoding usedMulti-Token Prediction used— | — | — |
| GR00T N1.7 core | Diffusion Transformer (DiT) coreFlow-matching transformer coreSemantically partitioned state and action vectors with field metadata used— | Cross-attention coreStrict action padding mask optionalStrict vision-language padding mask optional— | — | — | Adaptive Layer Normalization (AdaLN) core— | Diffusion Transformer IDM using SigLIP-2 vision embeddings used— | Cross-embodiment modeling coreDual-system compositional architecture coreEmbodiment-indexed proprioception MLP coreEmbodiment-specific state and action projectors corePer-embodiment action MLPs coreSigLIP2 pretrained vision transformer coreT5 pretrained text encoder coreVision-language model with diffusion-transformer action head coreMiddle-layer VLM representations for policy conditioning usedPre-trained vision and language encoders used— | — |
| Gemma 4 31B core | Decoder-only Transformer architecture corePer-Layer Embeddings (PLE) coreDense model architecture used— | Hybrid Sliding-Window and Global Attention coreUnified Keys and Values core— | Mixture of experts coreSparse expert activation core— | 2D coordinate-based positional embeddings usedAxial 2D-RoPE usedProportional Rotary Position Embedding used— | — | MTP speculative decoding core— | Encoder-free multimodal architecture coreDedicated multimodal encoders before the LLM usedFrozen encoders during pre-training usedLightweight vision embedding module usedRaw audio projection into LLM embedding space usedSingle-matmul vision projection usedUniversal Speech Model-based audio encoder usedVariable aspect-ratio image handling usedConfigurable visual token budget optional— | — |
| Inkling core | Regular Transformer decoder coreShort convolutions in attention and residual branches used— | Hybrid Sliding-Window and Global Attention coreGrouped-query attention usedKernel-4 convolutions in decoder blocks usedSliding Window Attention used— | Mixture of experts coreMoE with expert sinks coreShared expert coreTop-k expert routing coreSigmoid router with auxiliary-loss-free load balancing used— | Learned relative attention coreLearned input-dependent relative-position bias used— | Post-embedding RMSNorm used— | — | Discrete audio token encoding coreHierarchical patch encoder coreJoint multimodal decoding in a shared hidden space coreNative multimodal architecture coreEncoder-free multimodal architecture used— | — |
| Kimi K3 core | — | Gated Multi-head Latent Attention coreHybrid linear attention mechanism coreKDA coreKimi Delta Attention coreKimi Delta Attention–Attention Residuals architecture coreLayerwise Hybrid Linear and Global Attention coreFactorized spatial-temporal attention usedGated DeltaNet usedInput-dependent full-rank output gate usedLower-bounded scaled-sigmoid log-decay used— | Auxiliary-loss-free load balancing coreDispatch recomputation coreLatentMoE coreMixture of experts coreOutput-independent MoE gradient reformulation coreQuantile Balancing coreSharded latent weights with fused all-gather GEMM epilogue coreSiTU-GLU coreSparse expert activation coreStable LatentMoE coreTop-k routing with frozen bias defaultExact coordinate minimization usedExponential moving average of estimated quantiles usedHistogram-based global quantile estimation usedLatent-space routing usedPooled-global-batch quantile estimation usedSigmoid Tanh Unit (SiTU) usedUniform-bin histogram quantile estimation used— | No Position Encoding core— | Attention Residuals coreBlock Attention Residuals coreTwo-phase Block AttnRes schedule coreFull Attention Residuals mentioned— | — | Native multimodal architecture coreDecoupled Encoder Process (DEP) usedDual-format coordinate supervision usedFrom-scratch MoonViT-V2 training with next-token prediction usedMoonViT-V2 usedMultimodal input usedPixelShuffle 2×2 visual-token downsampling used— | Progressive context extension used— |
| Kolibri-1 core | Residual-stream width selection evaluated— | Full attention without positional encoding coreHybrid Sliding-Window and Global Attention coreSliding Window Attention core— | Exact Quantile Balancing coreLoad-error gradient injection coreMixture of experts coreShared expert coreTop-k expert routing coreMany small experts usedMixture-of-experts load balancing usedRecentered global-quantile target update usedTwo-pass radix selection usedCoarse-grained MoE configuration evaluatedExpert granularity evaluatedFine-grained MoE configuration evaluatedRandom routed-expert selection evaluatedRouted-expert silencing test evaluatedLeading dense feed-forward layers before MoE layers not usedMoE layer densification by expert concatenation not used— | No Position Encoding coreStatic YaRN used— | QK Normalization coreQK-Clip evaluated— | — | — | Context-length extrapolation without positional rescaling used— |
| LFM3-1.7B core | Hybrid architecture core— | Gated short convolution coreGlobal attention coreGrouped-query attention coreInput-dependent multiplicative gating around a depthwise short convolution coreKernel-size-7 convolution coreSliding Window Attention coreSparse use of grouped-query attention blocks for global context core— | Mixture of experts coreTop-k routing with normalized sigmoid and adaptive load-balancing bias coreDense-first-layer, sparse-MoE-later-layer placement used— | Full RoPE in sliding-window attention layers corePartial rotary position embeddings core— | Head-wise gated attention coreSandwich normalization core— | Residual-MLP action head for single-pass prediction of eight-step action chunks core— | Causal audio encoder with sliding-window attention and short convolutions coreConvolution-attention vision encoder coreDiscrete Residual Vector Quantization audio generation coreDual proprioception conditioning coreEight-codebook Mimi RVQ codes corePairwise sampled-frame embedding coreThree-stage multimodal training protocol coreVision-language extension with feature connector coreASR-based connector alignment usedConnector pre-training with frozen encoders usedDynamic image tiling usedFixed 6:12 text-to-audio token interleaving usedParallel action prediction usedPixelUnshuffle visual-token downsampling usedRepurposed navigation vocabulary tokens usedResidual-quantized trajectory primitives usedSigLIP2 vision encoder with dynamic tiling and token reduction usedSingle-pass continuous action chunk prediction usedVisual route following with target and waypoint prediction usedGlobal thumbnail alongside tiled image inputs optionalVision-and-language navigation evaluated— | — |
| Laguna-S-2.1 core | — | Grouped-query attention coreHybrid Sliding-Window and Global Attention coreSoftplus attention gating coreGated DeltaNet and full attention usedSoftplus-based per-head attention gating usedDense gated attention with full RoPE and full gating evaluatedPer-Head Gating evaluatedSWA-1024 (3:1 Interleaving) evaluatedSWA-512 evaluated— | Mixture of experts coreSparse expert activation coreToken-choice routing with softplus gating coreAuxiliary-loss load balancing usedRouted expert modulation usedShared expert usedToken-choice routing used— | Per-layer-type rotary position scales coreRotary Position Embedding usedGated attention with partial RoPE evaluated— | — | — | — | — |
| LingBot-VLA 2.0 core | Mixture-of-Transformers (MoT) core— | Blockwise causal attention coreShared self-attention across modalities core— | Auxiliary-loss-free load balancing coreFine-grained expert segmentation coreFine-grained routed experts with a shared expert coreMixture of experts corePer-expert routing correction bias coreShared expert isolation coreSigmoid-based expert routing coreSign-based expert-load correction-bias updates core— | Shared spatial and modality-specific positional embeddings coreUV positional encodings core— | — | Diffusion-policy action head used— | ConvStack coreHierarchical convolutional decoder coreLearnable image queries corePerceptual-query token appending coreSeparate RGB and depth patch embeddings coreDINOv2 and Point Transformer grasp-policy conditioning usedDepth injection optional— | — |
| MiMo-V2.5 core | — | Global attention coreHybrid Sliding Window Attention coreHybrid Sliding-Window and Global Attention coreSliding Window Attention coreFull attention usedGrouped-query attention usedLearnable attention sink bias usedSWA-128 used— | Mixture of experts coreDense feed-forward network usedExpert bias update factor usedLoad balancing usedTop-k expert routing usedRound-robin load balancing optionalMoE without shared experts not used— | Rotary Position Embedding used— | — | Multi-Token Prediction coreMTP speculative decoding used— | Native omnimodal architecture coreNative visual and audio understanding coreDedicated visual and audio encoders with lightweight projectors usedMiMo-Audio-initialized audio encoder usedProjector warmup usedVision Transformer (ViT) used— | 1M-token context window usedProgressive context extension used— |
| MiMo-V2.5-Pro core | — | Hybrid Sliding-Window and Global Attention coreGlobal attention usedHybrid Sliding Window Attention usedLearnable attention sink bias usedSliding Window Attention used— | Mixture of experts core— | — | — | MTP speculative decoding usedMulti-Token Prediction used— | Native multimodal architecture coreNative omnimodal architecture coreNative visual and audio understanding coreDedicated visual and audio encoders with lightweight projectors usedProjector warmup used— | 1M-token context window usedProgressive context extension used— |
| MiMo-V2.6-Pro core | — | Alternating Row-Major and Column-Major Token Serialization coreHybrid Sliding-Window and Global Attention coreHybrid sparse mixture-of-experts Transformer architecture coreSink-Augmented Sliding Window Attention core— | Mixture of experts coreMoE without shared experts coreSparse expert activation core— | — | — | Multi-Token Prediction core— | Four-frame audio patches with within-patch bidirectional attention coreTwo-stage audio encoding with tokenization and patch encoding coreData-parallel-first multimodal encoding usedJoint full-model multimodal training usedTwo-stage text-then-multimodal pre-training used— | — |
| MiniMax-M3 core | — | Block-level selection coreBlock-sparse grouped-query attention coreDynamic sparse selection coreGrouped-query attention coreIndex Branch coreMain Branch coreMiniMax Sparse Attention coreSparse attention coreBlock Max Pooling usedIndexer warmup usedKV-outer sparse attention usedLocal Block usedNatively trained sparsity usedSingle-head index key usedSparse softmax attention usedTop-k block selection usedTwo-stage sparse-attention formulation usedDeepSeek Sparse Attention evaluatedFull attention evaluatedMixture of Block Attention evaluatedSliding Window Attention evaluatedLinear attention mentionedCompressed Sparse Attention not usedDense Attention Fallback unclearForced Sink and Local Window not usedIndex Branch output not usedIndex value head not usedLearnable attention sink token not usedLightning Attention not usedMulti-head Latent Attention not usedNative Sparse Attention not usedRoPE with attention sink unclear— | Mixture of experts corePersistent load balancing usedShared expert usedTop-k expert routing used— | Rotary Position Embedding used— | — | — | Mixed-modality training from the start core— | — |
| Mistral-Large-4 core | — | DeepSeek Sparse Attention indexer coreMulti-head Latent Attention coreMulti-head latent attention with a DSA sparse indexer coreSparse attention coreFlashMLA sparse attention defaultDeepSeek Sparse Attention usedKL-distillation indexer warm-up usedShared DSA indexer across sliding-window layers usedSSMax usedTop-2,048 key-token selection per query usedCross-layer attention sharing not usedRagged Multi-head Latent Attention not usedSliding Window Attention not used— | Granular Mixture-of-Experts with routed and shared experts coreHybrid instruction-and-reasoning MoE coreMixture of experts coreShared expert coreTop-k expert routing coreExpert-parallel-group load-balancing loss usedMoE granularity 4 usedLoad balancing mentioned— | Alternating RoPE/NoPE layers coreLeRoPE coreYaRN evaluated— | — | — | Multimodal input coreNative multimodal architecture corePixtral vision encoder corePixtral-style variable-resolution vision encoder coreSpatial patch-merge projector core— | — |
| MolmoAct 2 core | Flow-matching action expert core— | Attention-based pooling of local visual windows coreBidirectional self-attention with VLM cross-attention coreForward attention among visual tokens coreNoncausal transformer action expert coreWindowed multi-head attention pooling coreScaled Dot-Product Attention (SDPA) used— | — | Rotary Position Embedding core— | Learned per-layer depth gate coreDepth-conditioning gate initialization usedLearned depth gate on expert conditioning used— | Discrete autoregressive robot policy corePredict discrete depth tokens before action generation corePredicting end-effector trajectory traces as intermediate visual plans coreSequential depth, trajectory, and action token generation coreDiffusion-Transformer-style action expert used— | Continuous DiT-style action expert coreDepth-aware perception tokens coreMasking action tokens from expert conditioning coreMulti-layer ViT feature concatenation coreMulti-scale image cropping corePer-layer key-value conditioning coreSingle resized crop per image coreTime-conditioned DiT modulation coreGrid quantization of depth maps usedIndex-token marking for multi-image inputs usedLearned image-padding-status embeddings usedMolmo2-ER backbone usedNear-zero action-expert output initialization usedAutoregressive depth-token prefix conditioning optionalHidden-state conditioning evaluatedPer-head per-layer key-value conditioning evaluated— | — |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B core | — | Grouped dynamic convolution coreHybrid Mamba-2 and Mixture-of-Experts Architecture coreHybrid Mamba-Transformer coreMamba-2 coreSliding Window Attention with an 8,192-Position Window used— | LatentMoE coreMixture of experts coreSparse expert activation coreDisabled router-bias updates usedLatent-dimension reduction with increased expert count and active experts used— | No Position Encoding core— | — | Frozen-backbone MTP-head-only fine-tuning coreMulti-Token Prediction coreRecursively shared Multi-Token Prediction head coreArchitecture-aware MTP-head initialization by projection fusion and pruning usedHidden-state mixing during recursive MTP training usedMTP-head warm-up with a frozen backbone usedMTP-only training used— | — | — |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B core | Hybrid architecture core— | Grouped dynamic convolution coreHybrid Mamba-Transformer coreMamba checkpointing coreSliding Window Attention with an 8,192-Position Window used— | Mixture of experts coreDisabled router-bias updates used— | — | — | Frozen-backbone MTP-head-only fine-tuning coreMulti-Token Prediction coreRecursively shared Multi-Token Prediction head coreShared Multi-Token Prediction head coreArchitecture-aware MTP-head initialization by projection fusion and pruning usedHidden-state mixing during recursive MTP training usedMTP-head warm-up with a frozen backbone usedMTP-only training used— | C-RADIO vision encoder coreVision-language connector coreVideo-frame pruning used— | — |
| OpenVLA / OpenVLA-OFT core | Spatially shared FiLM modulation across vision-transformer patch embeddings used— | Bidirectional attention core— | Feature-wise Linear Modulation (FiLM) optional— | — | — | Empty action embeddings with distinct positional encodings coreParallel action decoding core— | Continuous action representation coreFused SigLIP-DINOv2 vision encoder coreMulti-view image input coreRepresenting robot actions as language-model tokens coreRobot proprioceptive-state projection into language embedding space coreFiLM usedMulti-resolution visual feature fusion usedPatch-as-token vision-language architecture usedShared-sequence multimodal input projection usedStatic third-person camera input usedSingle SigLIP vision encoder evaluated— | — |
| Qwen3.5-397B-A17B core | — | DeepSeek Sparse Attention coreGated attention with reduced KV heads coreGated DeltaNet coreGated DeltaNet and full attention coreGated DeltaNet–sparse Mixture-of-Experts hybrid coreLinear attention coreDynamic Attention Window Size Training usedFull attention with multi-head attention for reliability used— | Hybrid Mixture of Experts coreMixture of experts coreMoE with 10 routed and 1 shared expert activated per token coreMoE with 8 routed and 1 shared expert activated per token coreSparse expert activation core— | Static YaRN usedRoPE scaling optionalRoPE scaling for context extension (YaRN) optionalYaRN optionalYaRN configuration optionalYaRN scaling-factor selection optionalTemporal-Modality Rotary Position Embedding not used— | — | Multi-Token Prediction usedMulti-Token Prediction module for residual codebooks used— | Early-fusion multimodal training coreThinker–Talker architecture core128-channel mel-spectrogram audio front end usedAudio Transformer (AuT) usedCausal streaming ConvNet codec decoder usedExplicit text-string timestamps usedRVQ-based speech representation usedVideo preprocessor longest-edge configuration optional— | — |
| Qwen3.6-35B-A3B core | — | Gated DeltaNet coreGated DeltaNet and full attention coreGated DeltaNet hybrid attention core— | Gated DeltaNet MoE coreMixture of experts core— | RoPE scaling optionalYaRN optionalYaRN scaling-factor selection optional— | — | — | Early-fusion multimodal training core— | — |
| Qwen3.8-Flash-Next core | — | Compressed lightweight indexer coreFixed-budget sparse-attention selection coreGated DeltaNet coreGated DeltaNet and full attention coreMQA indexer coreQSA micro-block compression at ratio 4 coreQwen Sparse Attention coreReuse QSA index selection across speculative decoding steps coreTop-512 block selection coreAverage pooling usedBlock-causal scoring usedBounded sigmoid output gate usedJoint backbone and indexer training under sparse attention usedTop-scoring compressed-block selection used— | Mixture of experts coreMoE with 10 routed and 1 shared expert activated per token coreContextual gating for N-gram embedding injection usedFrequency-based partitioning of N-gram embedding slots evaluatedSwiGLU-clip not used— | Partial rotary position embeddings usedStatic YaRN usedYaRN optional— | All-branch, elementwise read gating in Gated Residual coreDynamic gating of residual reads and writes coreElement-wise data-dependent residual read gate coreFour-branch residual stream coreFour-stream Hyper-Connection Combine update coreGated Residual coreGatedNorm coreIndependent per-branch normalization in Gated Residual coreLow-rank gated Mix for Hyper-Connection residual streams corePer-branch scalar residual write gate coreWidened residual stream coreBounded positive gates usedData-dependent residual read and write operators usedZero-centered RMSNorm usedHyper-Connections evaluatedManifold-Constrained Hyper-Connections evaluatedSimplified AltUp variant evaluatedQK-Clip not usedSparse gated residual writes not used— | Multi-Token Prediction core— | — | Host-memory-prefetched N-gram embedding tables coreN-gram embedding lookup memory coreN-gram embedding table for parameter scaling coreSingle-layer N-gram embedding placement coreAsynchronous prefetching of offloaded N-gram embeddings usedHost-memory offload for N-gram embeddings usedMulti-head hashing for N-gram embedding lookup usedToken normalization for N-gram vocabulary compression evaluated— |
| SmolVLA core | Flow-matching action expert core— | Causal self-attention coreCross-attention coreInterleaved Self-Attention and Cross-Attention coreCausal attention usedBidirectional attention evaluatedCross-attention-only action expert evaluated— | Action expert core— | — | — | — | Intermediate VLM-layer features coreReduced action-expert hidden size coreSmall pretrained VLM backbone coreVision-language model backbone coreVisual token reduction coreGlobal-image-only runtime input defaultIntermediate-layer-limited action-expert conditioning defaultSensorimotor-state projection into VLM token space evaluated— | — |
| Step-3.7-Flash core | — | Cascade attention not used— | Mixture of experts core— | — | — | MTP speculative decoding optionalMulti-Token Prediction optional— | Language backbone coreVision encoder coreVision Transformer (ViT) used— | 256K context window core— |
| VLA Foundry (Foundry-VLA-1.7B) core | — | — | — | — | — | Flow-matching action head coreDiffusion-policy action head optionalAutoregressive discrete action-tokenization head mentioned— | Last-four-layer action-head conditioning corePretrained Qwen3-VL backbone initialization used— | — |
| X-VLA core | Flow-based vision-language-action model core— | — | Parallel MLP-Mixer-based action decoder evaluatedHeterogeneity-guided MoE routing not usedLoad-balancing regularization not used— | — | — | Separate embodiment-specific action projection heads evaluated— | Per-hardware-configuration learnable prompt embeddings coreStreamlined encoding pipeline with soft prompts coreCross-attention-based resampler usedDomain-specific resamplers and action heads usedEmbodiment-aware language prompt concatenation usedPretrained vision-language encoder usedShared vision backbone usedAction-modality isolation evaluatedHPT-style domain-specific input projection evaluatedMultimodal Diffusion Transformer decoder evaluatedNatural-language hardware-configuration prompts evaluated— | — |
| Xiaomi-Robotics-1 core | Mixture-of-Transformers (MoT) core— | Causal attention usedLearnable attention sink token usedShape attention mask for action-prefix conditioning used— | — | RoPE positional-index offset used— | Adaptive Layer Normalization (AdaLN) used— | — | Masking action-related tokens from DiT attention used— | — |
| gpt-oss-120b core | — | Grouped-query attention usedHybrid Sliding-Window and Global Attention usedOff-by-one attention used— | Mixture of experts coreTop-k routing with softmax over selected experts coreSwiGLU used— | Rotary Position Embedding usedYaRN used— | Pre-LN usedRMSNorm used— | — | — | — |
| π0.5 core | Flow-based vision-language-action model coreUnified model for high-level and low-level inference coreDownsized action expert usedSemantic action representations using language sub-tasks mentioned— | Bidirectional attention coreThree-block causal attention mask corePrefix-conditioned autoregressive action-token attention usedUnidirectional VLM-to-action-expert information flow used— | Action expert core— | — | Adaptive RMSNorm used— | Flow-matching action head defaultDiffusion decoding for vision-language-action policies evaluatedRegression-head action prediction mentioned— | Routing image and language tokens to the VLM backbone coreRouting robotics-specific inputs to an action expert coreSeparate flow-matching action expert coreMasking missing robot image slots usedSeparate robotics-specific token weights used— | — |