Allen Institute for AI
MolmoAct 2
What MolmoAct 2’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 5.4B
- Attention
- Molmo-family VLM; flow-matching action head (v2); weights, code and data open
- Input
- text+image -> action
- Licence
- Apache-2.0 (code); card has no license tag: check
- Released
- 2026-05-04
Against the consensus recipe
Its documents state 1 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationused
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertsnot in its documents
- Multi-Token Predictionnot in its documents
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modenot in its documents
- Multi-Token Predictionnot in its documents
- FP8not in its documents
Checked in its code
Its code shows 3 of the 19 architecture features checked — 3 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from the model repository’s own code (modeling_molmoact2.py, configuration_molmoact2.py, inference.py) and config.json at revision e432d85f6e, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention code only | in its code | not stated | The text model's attention uses 32 query heads with 8 KV heads, repeated via repeat_kv for grouped-query attention. show evidencetext_config.num_attention_heads = 32text_config.num_key_value_heads = 8 |
| Multi-head latent attention | not in its code | not stated | There is no low-rank KV latent projection (kv_lora_rank/q_lora_rank) anywhere in the text model; attention uses standard fused QKV… show the whole verdictThere is no low-rank KV latent projection (kv_lora_rank/q_lora_rank) anywhere in the text model; attention uses standard fused QKV projection instead. |
| Sliding-window attention | not in its code | not stated | No sliding_window, use_sliding_window, or layer_types config keys exist and the attention implementation builds a standard causal mask… show the whole verdictNo sliding_window, use_sliding_window, or layer_types config keys exist and the attention implementation builds a standard causal mask with no windowing logic. |
| Interleaved sliding-window and global attention | not in its code | not stated | No layer_types or sliding_window_pattern exist in config and all decoder layers use the same MolmoAct2Attention class uniformly, so there… show the whole verdictNo layer_types or sliding_window_pattern exist in config and all decoder layers use the same MolmoAct2Attention class uniformly, so there is no interleaving of sliding and full attention. |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | not in its code | not stated | No indexer scoring mechanism or index_topk/index_n_heads/index_head_dim config keys exist; attention is standard dense softmax attention. |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | All text decoder layers use the same MolmoAct2Attention softmax-attention class; there is no linear-attention or state-space mixer layer… show the whole verdictAll text decoder layers use the same MolmoAct2Attention softmax-attention class; there is no linear-attention or state-space mixer layer type in the code. |
| Gated DeltaNet layers | not in its code | not stated | No gated delta rule or recurrent state update mechanism exists in the text model code; attention layers are standard softmax attention. |
| Mamba-2 layers | not in its code | not stated | No Mamba-2/SSD selective state-space block implementation or related config keys (mamba_num_heads, ssm_state_size) are present. |
| Learnable attention sink | not in its code | not stated | No learned per-head sink logit is added to the attention softmax denominator anywhere in the attention forward implementation. |
| Gated attention output | not in its code | not stated | The attention output is passed directly to attn_out with no sigmoid gating multiplication applied to it in MolmoAct2Attention.forward. |
| QK normalization code only | in its code | not stated | use_qk_norm is true and qk_norm_type is 'qwen3', causing MolmoAct2RMSNorm to be applied per-head to queries and keys before the dot product. show evidencetext_config.use_qk_norm = truetext_config.qk_norm_type = "qwen3" |
| Partial RoPE | not in its code | not stated | RoPE is applied to the full head_dim (rotate_half splits the whole dimension in half for rotation, not a fractional sub-range), and there… show evidenceRoPE is applied to the full head_dim (rotate_half splits the whole dimension in half for rotation, not a fractional sub-range), and there is no partial_rotary_factor or qk_rope_head_dim config key. |
| YaRN RoPE scaling | not in its code | not stated | rope_parameters.rope_type is 'default' (no YaRN scaling type set), so the RoPE scaling code path for YaRN is not switched on. |
| Layers without positional encoding (NoPE) | not in its code | not stated | Every decoder layer calls apply_rotary_pos_emb on position_embeddings unconditionally; rope_scaling_layers is null so there's no per-layer… show evidenceEvery decoder layer calls apply_rotary_pos_emb on position_embeddings unconditionally; rope_scaling_layers is null so there's no per-layer selection skipping RoPE on some attention layers. text_config.rope_scaling_layers = null |
| Mixture of experts | not in its code | not stated | The feed-forward module (LanguageModelMLP) is a single dense SwiGLU MLP per layer with no router or expert selection mechanism, and no… show evidenceThe feed-forward module (LanguageModelMLP) is a single dense SwiGLU MLP per layer with no router or expert selection mechanism, and no num_experts/num_experts_per_tok config keys exist. |
| Shared expert | not in its code | not stated | There is no mixture-of-experts implementation in the text model, hence no shared-expert mechanism either; no n_shared_experts config key… show the whole verdictThere is no mixture-of-experts implementation in the text model, hence no shared-expert mechanism either; no n_shared_experts config key exists. |
| Auxiliary-loss-free load balancing (selection bias) | not in its code | not stated | No router or e_score_correction_bias mechanism exists since the model has no MoE layers. |
| Multi-token prediction layers | not in its code | not stated | No num_nextn_predict_layers/mtp_num_layers config key or extra next-next-token prediction module exists in the code; the model only has a… show the whole verdictNo num_nextn_predict_layers/mtp_num_layers config key or extra next-next-token prediction module exists in the code; the model only has a single lm_head for next-token prediction. |
| SwiGLU feed-forward code only | in its code | not stated | LanguageModelMLP computes SiLU(gate) * x with hidden_act='silu', matching SwiGLU for every decoder layer's feed-forward. show evidencetext_config.hidden_act = "silu" |
filed at the root 1
“Post-training therefore adds a flow-matching action expert to the pre-trained VLM, producing the final MolmoAct2 model.”
token mixer 6
“Each block contains bidirectional self-attention over the action chunk, cross-attention to the VLM context, and an MLP.”
“Visual tokens are allowed to forward-attend to one another, including across different frames or camera views”
“The action expert is a noncausal transformer over a short action chunk.”
“within each patch window, a multi-headed attention layer pools the four patches to a single vector, using the mean of the patches as the…”
softmax attention 2
“then pools local windows with a multi-head attention layer.”
“use PyTorch’s scaled dot-product attention implementation rather than FlashAttention because our packed multimodal sequences and…”
positional encoding 1
“The expert uses rotary position embeddings for the action sequence, which gives the denoising transformer an explicit ordering over the…”
normalization & residual 3
“The baseline uses both 10% depth-token noise and the learned per-layer depth gate described in Sec. 5.2”
“The gate is initialized with bias , so fine-tuning begins close to the standard action-conditioning path”
“The LIBERO depth fine-tuned model also includes a learned depth gate on the expert conditioning path.”
prediction head 5
“MolmoAct2-Pretrain adapts the Molmo2-ER vision-language backbone into a discrete autoregressive robot policy.”
“before producing an action, the model predicts a compact discrete depth representation that conditions the action expert through per-layer…”
“we generate the 2D trajectory of the end-effector and train the model to predict this trajectory alongside the next action command”
“the model autoregressively generates three token sequences in order: (i) depth perception tokens; (ii) visual reasoning trace; (iii)…”
“We use a DiT-style transformer expert because flow matching and diffusion models have shown that denoising transformers scale well for…”
multimodal architecture 14
“connects the autoregressive VLM to a flow-matching continuous action expert through per-layer KV (key-value) conditioning.”
“post-training attaches a continuous DiT-style action expert to this already robot-aware backbone.”
“By combining depth perception tokens, visual reasoning traces, and action prediction”
“The discrete action-token span is masked from the expert conditioning path”
“features from the third-to-last (OpenAI CLIP) or fourth-to-last (SigLIP2) and the tenth-from-last ViT layers are concatenated for each patch”
“tiling each image with multiple square high-resolution crops plus a resized low-resolution full image”
“each camera observation is represented as a single resized crop rather than a tiled high-resolution image.”
“A sinusoidal time embedding is passed through a small MLP, then used to produce DiT-style shift, scale, and gate parameters for all three…”
“Each observation depth map is quantized into a grid, giving 100 spatial code positions, and each position takes one of 128 learned…”
“we append index tokens at the beginning of the vision tokens of each image, and we finally concatenate all images together as the input.”
“A learned embedding indicating the crop’s padding status (no padding, some padding, or all padding) is added to the patch features”
“We directly fine-tune Molmo2 and Molmo2-ER with the same discrete-action architecture”
“The final projection is initialized to produce near-zero velocity predictions at the beginning of post-training”
“MolmoAct2-Think keeps the same architecture and adds an autoregressive depth-token prefix before action prediction.”
2 more mentioned or evaluated
multimodal architecture “We compare hidden-state conditioning, standard per-layer KV conditioning, and a per-head per-layer KV conditioning variant.”
multimodal architecture “In the per-head variant, the VLM keys and values remain separated by head, and each head is projected into the corresponding action-expert…”
filed at the root 3
“Overall, full-model adaptation with discrete and continuous action co-training provides the best average outcome.”
“We use , so each robot sample contributes four points on the same flow trajectory while reusing the same visual-language context.”
“The VQVAE is trained with a standard reconstruction objective to minimize reconstruction loss between input RGB images and their…”
language modelling objective 3
“the loss is computed only on the action tokens”
“adds robot-specific state, setup/control, and action-output tokens so that robot trajectories can be trained with the same next-token…”
“Pre-training first turns Molmo2-ER into an action-aware VLA using only autoregressive supervision”
distillation objective 1
“the VLA is trained to predict this string autoregressively from the original RGB observation”
auxiliary loss 3
“The expert conditions on the VLM keys and values, but these tensors are detached before they enter the expert, so updates the action…”
“their text, visual tokens, state tokens, and action targets remain separated by the training attention mask, so the model does not…”
“10% of depth-code input tokens are replaced by uniformly sampled depth codes, while the prediction targets remain unchanged.”
filed at the root 1
“we follow the training scheme of LLaVA-AURORA (Bigverdi et al., 2025) by unfreezing the tokenizer embedding and lm head.”
training precision 2
“We use automatic mixed precision with bfloat16 for most operations, while keeping numerically sensitive operations such as layer…”
“As an exception, we compute layer normalization and Rotary Position Embeddings (RoPE) in fp32.”
training parallelism 4
“We train in PyTorch using Fully Sharded Data Parallel (FSDP)”
“each device divides by the average number of loss tokens across all devices rather than by its local number of loss tokens.”
“we register a gradient hook on those tensors to synchronize their gradients across ranks before the optimizer step.”
“we do not shard LoRA adapter parameters under FSDP. Instead, each GPU keeps a full copy of the LoRA parameters”
data sourcing 2
“for each of the cameras in the trajectory that are not frozen, we sample 12 frames that are each equally temporally spaced”
“We isolated and separately collected data for each atomic motion—open, pick, flip, put, and close—to build a comprehensive set of motion…”
data filtering 10
“Video examples are sampled at up to 2 FPS and capped to at most 8 frames.”
“we trained on a modified dataset that filtered out no-op actions and unsuccessful demonstrations”
“removal of eval-style datasets”
“we apply a four-stage filtering pipeline: (i) structural validity checks”
“the provided idle-frame filter, which retains only contiguous non-idle action segments of at least one second.”
“license/codebase eligibility checks”
“raw task strings are normalized into concise natural-language task descriptions.”
“required schema fields, valid action/state tensors, no NaN/corrupt samples”
“we keep datasets whose mean TOPReward over the last 3 sampled episodes is above a threshold obtained by averaging the TOPReward over a…”
“we use TOPReward (Chen et al., 2026) to filter out 438 low-quality dataset entries”
synthetic data 4
“we collected half of the number of demonstrations in the regular way and the other half exploring alternative paths towards the same…”
“To increase the diversity of prompts, we randomly sample a number and ask the VLM to make this instruction roughly that many words.”
“In contrast, VLM-based point annotations (e.g., Molmo (Deitke et al., 2024), RoboPoint (Yuan et al., 2024)) yield accurate, non-degenerate…”
“we use an open-source VLM (Qwen3.5-27B) to re-annotate the datasets.”
data mixture & curriculum 13
“while uniformly mixing action-only and depth-and-action examples.”
“MolmoAct is pre-trained on a mixture of action reasoning data, auxiliary robot data, and multimodal web data.”
“data is sampled at the following rates: RT-1 (20%), BridgeData V2 (12.5%), BC-Z (7.5%)”
“We include a mixture of multimodal web data from Molmo’s Supervised fine-tuning stage”
“For multi-camera robot episodes, we randomize the input camera order at the episode level to prevent the model from relying on a fixed…”
“the tokenizer is trained on a heterogeneous mix of "dialects," including Absolute Joint positions and Delta End-Effector velocities.”
“All eight tasks are trained jointly in a multi-task setup, with demonstrations from every task and variation pooled into a single training…”
“on a mixture that interleaves our embodied corpus with Molmo2’s original multimodal mid-training data”
“Batches mix examples from multiple tasks.”
“by pairing each side view with the wrist view”
“MolmoAct2-FAST Tokenizer is distinguished by its use of a fully transparent training mix”
“Within robot data, post-training uniformly samples three tasks: action prediction, depth prediction, and depth-and-action prediction.”
“During fine-tuning, dataset mixing is performed within each batch so that each update contains examples from multiple sources.”
sequence packing 5
“Once the pool is full, a dynamic-programming solver selects the subset that maximizes”
“we pad the packed action-chunk axis up to a small fixed cap of five chunks per packed sequence when possible.”
“We use on-the-fly packing to combine multiple short examples into one 4200-token sequence.”
“In practice, we solve a quantized version of the packing problem by rounding token counts to the nearest multiple of 32.”
“each action chunk is paired with the sub-example that produced it, and the expert attends only to that example’s VLM context.”
tokenization 9
“we normalize using dataset quantiles and discretize into 256 uniform-width bins between the first and ninety-ninth percentiles”
“by adding robot-specific tokens to the backbone vocabulary.”
“We then assign them monotonically to the 256 bins so that adjacent bins map to adjacent symbols”
“While we utilize the core logic of frequency-domain compression (DCT-BPE)”
“representing the trajectory with a frequency-domain transform, quantizing the resulting coefficients, and applying byte-pair encoding to…”
“Continuous robot states are normalized using dataset statistics and discretized into one of 256 state tokens per scalar dimension.”
“Continuous actions are normalized, padded to a maximum width of 32 dimensions, and encoded by MolmoAct2-FAST Tokenizer into discrete…”
“we first pad the tokenizer and lm head to its next multiple of 512, then replace the first 130 tokens with our depth perception tokens”
“Robot examples are formatted with one of three output styles.”
filed at the root 4
“Post-training then attaches the continuous action expert to this already action-aware backbone”
“MolmoAct2 follows a three-stage training pipeline”
“This checkpoint is the post-trained, multi-embodiment MolmoAct2 model.”
“we build on the released Molmo2 checkpoint with a two-stage specialize-then-rehearse recipe.”
supervised fine-tuning 10
“Overall, full-model adaptation with discrete and continuous action co-training provides the best average outcome.”
“fine-tuning adapts that model quickly to a specific embodiment, environment, and use case”
“The baseline uses both 10% depth-token noise and the learned per-layer depth gate”
“We fine-tune MolmoAct-7B-D using Low-Rank Adaptation (LoRA)”
“we fine-tune for K steps on the Molmo2-ER corpus augmented with Tulu-3 (Lambert et al., 2025) text-only data to preserve language competence”
“deployment on robots outside this set ... requires effective fine-tuning on target-embodiment demonstrations”
“The MolmoAct2-LIBERO checkpoint is fine-tuned on the full LIBERO training mixture, combining the Spatial, Object, Goal, and Long suites…”
“each run is robot-only: we do not include the multimodal VLM mixture or a separate VLM dataloader.”
“We trained MolmoAct2-Post as eight separate single-task policies”
“Use this checkpoint for further fine-tuning on a target robot embodiment or benchmark.”
mid-training & continual pretraining 2
“we conduct a second stage of mid-training using high-quality action reasoning data closely aligned with our target domain of household…”
“MolmoAct-7B-D outperforms its counterpart without mid-training by an average margin of 5.5% across the three tasks, demonstrating that…”
1 more mentioned or evaluated
supervised fine-tuning “Tuning only the action expert is the clearest failure mode, dropping the average to 93.05%.”
Not used 2
supervised fine-tuning “we do not use knowledge insulation during fine-tuning: gradients from the flow loss are allowed to update the VLM through the…”
supervised fine-tuning “we do not tune the added-token input embeddings.”
filed at the root 2
“Where possible, actions should be monitored through interpretable intermediate outputs (adaptive depth map), simulation rollouts, action…”
“an optimized eager path using reusable-cache optimizations”
decoding strategy 2
“execute full chunks before redoing action reasoning”
“MolmoAct2 also exposes a discrete action path, in which the VLM autoregressively decodes action tokens for the entire action chunk.”
reasoning control 1
“new depth tokens are computed only for novel visual signals”
KV cache management 2
“Unchanged cells are replayed from the previous predicted depth buffer and consumed by the model as known depth-token inputs.”
“using a preallocated static KV cache to make the decode state stable across steps.”
inference scheduling 3
“We therefore cache reusable action-expert intermediates, including context-dependent cross-attention states and fixed position-dependent…”
“We therefore keep the adaptive scheduler eager, including span-level replay for unchanged cells, while using a preallocated static KV…”
“We use torch.compile for throughput and keep the ViT and LLM shapes static through padding and fixed sequence budgets”
inference kernel 1
“We further capture the fixed-shape flow loop with CUDA Graphs, reducing Python and kernel-launch overhead during repeated action generation.”
context management 2
“Instead of re-predicting every depth code at every step, MolmoAct2-Think reuses cached codes for static regions and autoregressively…”
“each patch is compared to the corresponding patch in the previous frame by cosine similarity. A cell is marked updated when the similarity…”
agentic scaffolding 7
“we allow the user to draw a visual reasoning trace directly on the camera image to indicate the desired end-effector path”
“open-ended natural language instructions provided interactively by participants”
“During inference, we provide the same formatted user message, assistant prefix, and trigger as the generation prefix”
“Setup and control descriptors make embodiment and action semantics explicit in the prompt.”
“For each observation frame, we prompt Molmo with the instruction "point to the robot gripper"”
“a web interface that enables users to modify the language instruction or sketch five points on the image for visual trace steering at test…”
“The behaviors can be executed directly, or steered via trajectory editing.”
filed at the root 2
“for evaluation, we select the best-performing checkpoint, which occurs at 40K updates.”
“In each efficiency fine-tuning task evaluation, we pre-marked the locations of all task objects in the scene to ensure that the evaluation…”
other 1
filed at the root 1
“This decomposition allows the policy to learn individual skill components in isolation before composing them into more complex behaviors.”
unfiled 31
“we apply action chunking (Zhao et al., 2023) during post-training, formatting action predictions in fixed-length chunks”
“MolmoAct frames visuomotor control as an autoregressive sequence modeling problem augmented with an action chain-of-thought (CoT)”
“MolmoAct2 predicts fixed-horizon action chunks (30 steps at 30 Hz for YAM/SO-100/101, 15 at 15 Hz for DROID, 10 at 10 Hz for LIBERO) and…”
“learns a flow-matching velocity field over continuous robot action chunks.”
“MolmoAct2-Think precedes this continuous action generation with an adaptive generation of discrete depth tokens only for dynamic scene…”
“The post-training objective combines the autoregressive loss from pre-training with the flow-matching loss”
“We adapt MolmoAct to post-train on single or multi-tasks via parameter-efficient LoRA fine-tuning.”
“executes them open-loop before re-querying the policy.”
“Post-training then attaches a flow-matching action expert with per-layer KV conditioning on the autoregressive backbone”
“MolmoAct grounds every step of its reasoning chain directly in the scene”
“integrates perception, planning, and control through a structured reasoning pipeline”
“By combining depth perception tokens, visual reasoning traces, and action prediction”
“We therefore use the continuous path as the default deployment option.”
“The grid squares are then moved closer together to introduce a fixed overlap margin (default 4 patches, approximately 56 pixels), which…”
“Continuous dimensions are normalized with 1–99 percentile statistics, which limits the effect of outliers while preserving the useful…”
“The image augmentation applies the stochastic transforms before image normalization and final resizing.”
“The model’s action space should be bounded by the training data, robot controller limits, and task-specific safety constraints, including…”
“Practically, this means inducing the robot to explore motion variants while still completing tasks, so the model learns rich…”
“To match OXE during pre-training, we downscale our collected images from 640480 to 320240 px while preserving the original aspect ratio.”
“At inference, we initialize the trajectory from Gaussian noise and integrate the learned velocity field for a fixed number of Euler steps”
“In post-training, we evaluate multiple sampled flow-matching times for each robot action chunk while reusing the same VLM context.”
“we set action chunk size to for evaluation on each task suites and execute full chunks before replanning.”
“we divide the device’s total loss by the global average number of loss-tokens per example across all devices.”
“the dimension mask zeroes padded action dimensions in the noisy inputs, targets, and predictions before averaging the loss over valid…”
“The horizon mask removes padded time steps from the flow loss”
“operate the system only in a safely configured environment with human supervision.”
“we simply replace the first 130 padding tokens with the depth perception tokens”
“Gripper commands are handled separately from this continuous normalization, since they are typically binary or narrow-range open/close…”
“each state value is uniformly discretized into one of 256 state tokens”
“The codebook is produced by a pre-trained specialist depth estimator”
“actions should be monitored through interpretable intermediate outputs (adaptive depth map), simulation rollouts, action limits”
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| huggingface.co/allenai/MolmoAct2 | model card | Ai2 | read |
| arxiv.org/abs/2605.02881 | technical report | Ai2 | read |
| arxiv.org/abs/2508.07917 | technical report | Ai2 | read |