Open Models Almanac
ModelsMolmoAct 2

Allen Institute for AI

MolmoAct 2

What MolmoAct 2’s own documents say it is built from — every method with the sentence that describes it.

Parameters
5.4B
Attention
Molmo-family VLM; flow-matching action head (v2); weights, code and data open
Input
text+image -> action
Licence
Apache-2.0 (code); card has no license tag: check
Released
2026-05-04

Against the consensus recipe

Its documents state 1 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 3 of the 19 architecture features checked — 3 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from the model repository’s own code (modeling_molmoact2.py, configuration_molmoact2.py, inference.py) and config.json at revision e432d85f6e, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention code only in its codenot stated

The text model's attention uses 32 query heads with 8 KV heads, repeated via repeat_kv for grouped-query attention.

show evidencetext_config.num_attention_heads = 32text_config.num_key_value_heads = 8
self.num_key_value_groups = (
            config.num_attention_heads // config.num_key_value_heads
        )
modeling_molmoact2.py · L2202–2204
key_states = repeat_kv(key_states, self.num_key_value_groups)
            value_states = repeat_kv(value_states, self.num_key_value_groups)
modeling_molmoact2.py · L2301–2302
Multi-head latent attention not in its codenot stated

There is no low-rank KV latent projection (kv_lora_rank/q_lora_rank) anywhere in the text model; attention uses standard fused QKV…

show the whole verdict

There is no low-rank KV latent projection (kv_lora_rank/q_lora_rank) anywhere in the text model; attention uses standard fused QKV projection instead.

Sliding-window attention not in its codenot stated

No sliding_window, use_sliding_window, or layer_types config keys exist and the attention implementation builds a standard causal mask…

show the whole verdict

No sliding_window, use_sliding_window, or layer_types config keys exist and the attention implementation builds a standard causal mask with no windowing logic.

Interleaved sliding-window and global attention not in its codenot stated

No layer_types or sliding_window_pattern exist in config and all decoder layers use the same MolmoAct2Attention class uniformly, so there…

show the whole verdict

No layer_types or sliding_window_pattern exist in config and all decoder layers use the same MolmoAct2Attention class uniformly, so there is no interleaving of sliding and full attention.

Indexer-selected sparse attention (DeepSeek Sparse Attention) not in its codenot stated

No indexer scoring mechanism or index_topk/index_n_heads/index_head_dim config keys exist; attention is standard dense softmax attention.

Linear-attention or state-space layers alongside full attention not in its codenot stated

All text decoder layers use the same MolmoAct2Attention softmax-attention class; there is no linear-attention or state-space mixer layer…

show the whole verdict

All text decoder layers use the same MolmoAct2Attention softmax-attention class; there is no linear-attention or state-space mixer layer type in the code.

Gated DeltaNet layers not in its codenot stated

No gated delta rule or recurrent state update mechanism exists in the text model code; attention layers are standard softmax attention.

Mamba-2 layers not in its codenot stated

No Mamba-2/SSD selective state-space block implementation or related config keys (mamba_num_heads, ssm_state_size) are present.

Learnable attention sink not in its codenot stated

No learned per-head sink logit is added to the attention softmax denominator anywhere in the attention forward implementation.

Gated attention output not in its codenot stated

The attention output is passed directly to attn_out with no sigmoid gating multiplication applied to it in MolmoAct2Attention.forward.

QK normalization code only in its codenot stated

use_qk_norm is true and qk_norm_type is 'qwen3', causing MolmoAct2RMSNorm to be applied per-head to queries and keys before the dot product.

show evidencetext_config.use_qk_norm = truetext_config.qk_norm_type = "qwen3"
if config.use_qk_norm:
            k_norm_size = (
                config.head_dim
                if config.qk_norm_type == "qwen3"
                else config.num_key_value_heads * config.head_dim
            )
            self.k_norm = MolmoAct2RMSNorm(k_norm_size, eps=config.layer_norm_eps)
modeling_molmoact2.py · L2224–2230
if (
            self.q_norm is not None
            and self.k_norm is not None
            and self.qk_norm_type == "qwen3"
        ):
            query_states = self.q_norm(query_states)
            key_states = self.k_norm(key_states)
modeling_molmoact2.py · L2274–2280
Partial RoPE not in its codenot stated

RoPE is applied to the full head_dim (rotate_half splits the whole dimension in half for rotation, not a fractional sub-range), and there…

show evidence

RoPE is applied to the full head_dim (rotate_half splits the whole dimension in half for rotation, not a fractional sub-range), and there is no partial_rotary_factor or qk_rope_head_dim config key.

def rotate_half(x):
    """Rotates half the hidden dims of the input."""
    x1 = x[..., : x.shape[-1] // 2]
    x2 = x[..., x.shape[-1] // 2 :]
    return torch.cat((-x2, x1), dim=-1)
modeling_molmoact2.py · L1934–1938
YaRN RoPE scaling not in its codenot stated

rope_parameters.rope_type is 'default' (no YaRN scaling type set), so the RoPE scaling code path for YaRN is not switched on.

show evidencetext_config.rope_parameters = {"rope_theta": 5000000.0, "rope_type": "default"}
Layers without positional encoding (NoPE) not in its codenot stated

Every decoder layer calls apply_rotary_pos_emb on position_embeddings unconditionally; rope_scaling_layers is null so there's no per-layer…

show evidence

Every decoder layer calls apply_rotary_pos_emb on position_embeddings unconditionally; rope_scaling_layers is null so there's no per-layer selection skipping RoPE on some attention layers.

text_config.rope_scaling_layers = null
cos, sin = position_embeddings
        query_states, key_states = apply_rotary_pos_emb(
            query_states, key_states, cos, sin
        )
modeling_molmoact2.py · L2285–2288
Mixture of experts not in its codenot stated

The feed-forward module (LanguageModelMLP) is a single dense SwiGLU MLP per layer with no router or expert selection mechanism, and no…

show evidence

The feed-forward module (LanguageModelMLP) is a single dense SwiGLU MLP per layer with no router or expert selection mechanism, and no num_experts/num_experts_per_tok config keys exist.

class LanguageModelMLP(nn.Module):
    def __init__(
        self,
        input_dim: int,
        intermediate_size: int,
        hidden_act: str,
        device: Union[str, torch.device] = None,
    ):
modeling_molmoact2.py · L2336–2343
Shared expert not in its codenot stated

There is no mixture-of-experts implementation in the text model, hence no shared-expert mechanism either; no n_shared_experts config key…

show the whole verdict

There is no mixture-of-experts implementation in the text model, hence no shared-expert mechanism either; no n_shared_experts config key exists.

Auxiliary-loss-free load balancing (selection bias) not in its codenot stated

No router or e_score_correction_bias mechanism exists since the model has no MoE layers.

Multi-token prediction layers not in its codenot stated

No num_nextn_predict_layers/mtp_num_layers config key or extra next-next-token prediction module exists in the code; the model only has a…

show the whole verdict

No num_nextn_predict_layers/mtp_num_layers config key or extra next-next-token prediction module exists in the code; the model only has a single lm_head for next-token prediction.

SwiGLU feed-forward code only in its codenot stated

LanguageModelMLP computes SiLU(gate) * x with hidden_act='silu', matching SwiGLU for every decoder layer's feed-forward.

show evidencetext_config.hidden_act = "silu"
def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.ff_proj(x)
        x, gate = x.chunk(2, dim=-1)
        x = self.act(gate) * x
        x = self.ff_out(x)
        return x
modeling_molmoact2.py · L2351–2356

filed at the root 1

Flow-matching action expert core

“Post-training therefore adds a flow-matching action expert to the pre-trained VLM, producing the final MolmoAct2 model.”

token mixer 6

Bidirectional self-attention with VLM cross-attention core

“Each block contains bidirectional self-attention over the action chunk, cross-attention to the VLM context, and an MLP.”

Forward attention among visual tokens core

“Visual tokens are allowed to forward-attend to one another, including across different frames or camera views”

Noncausal transformer action expert core

“The action expert is a noncausal transformer over a short action chunk.”

Windowed multi-head attention pooling core

“within each patch window, a multi-headed attention layer pools the four patches to a single vector, using the mean of the patches as the…”

softmax attention 2

Attention-based pooling of local visual windows core

“then pools local windows with a multi-head attention layer.”

Scaled Dot-Product Attention (SDPA) used

“use PyTorch’s scaled dot-product attention implementation rather than FlashAttention because our packed multimodal sequences and…”

positional encoding 1

Rotary Position Embedding core

“The expert uses rotary position embeddings for the action sequence, which gives the denoising transformer an explicit ordering over the…”

normalization & residual 3

Learned per-layer depth gate core

“The baseline uses both 10% depth-token noise and the learned per-layer depth gate described in Sec. 5.2”

Depth-conditioning gate initialization used

“The gate is initialized with bias , so fine-tuning begins close to the standard action-conditioning path”

Learned depth gate on expert conditioning used

“The LIBERO depth fine-tuned model also includes a learned depth gate on the expert conditioning path.”

prediction head 5

Discrete autoregressive robot policy core

“MolmoAct2-Pretrain adapts the Molmo2-ER vision-language backbone into a discrete autoregressive robot policy.”

Predict discrete depth tokens before action generation core

“before producing an action, the model predicts a compact discrete depth representation that conditions the action expert through per-layer…”

Predicting end-effector trajectory traces as intermediate visual plans core

“we generate the 2D trajectory of the end-effector and train the model to predict this trajectory alongside the next action command”

Sequential depth, trajectory, and action token generation core

“the model autoregressively generates three token sequences in order: (i) depth perception tokens; (ii) visual reasoning trace; (iii)…”

Diffusion-Transformer-style action expert used

“We use a DiT-style transformer expert because flow matching and diffusion models have shown that denoising transformers scale well for…”

multimodal architecture 14

Per-layer key-value conditioning core

“connects the autoregressive VLM to a flow-matching continuous action expert through per-layer KV (key-value) conditioning.”

Continuous DiT-style action expert core

“post-training attaches a continuous DiT-style action expert to this already robot-aware backbone.”

Depth-aware perception tokens core

“By combining depth perception tokens, visual reasoning traces, and action prediction”

Masking action tokens from expert conditioning core

“The discrete action-token span is masked from the expert conditioning path”

Multi-layer ViT feature concatenation core

“features from the third-to-last (OpenAI CLIP) or fourth-to-last (SigLIP2) and the tenth-from-last ViT layers are concatenated for each patch”

Multi-scale image cropping core

“tiling each image with multiple square high-resolution crops plus a resized low-resolution full image”

Single resized crop per image core

“each camera observation is represented as a single resized crop rather than a tiled high-resolution image.”

Time-conditioned DiT modulation core

“A sinusoidal time embedding is passed through a small MLP, then used to produce DiT-style shift, scale, and gate parameters for all three…”

Grid quantization of depth maps used

“Each observation depth map is quantized into a grid, giving 100 spatial code positions, and each position takes one of 128 learned…”

Index-token marking for multi-image inputs used

“we append index tokens at the beginning of the vision tokens of each image, and we finally concatenate all images together as the input.”

Learned image-padding-status embeddings used

“A learned embedding indicating the crop’s padding status (no padding, some padding, or all padding) is added to the patch features”

Molmo2-ER backbone used

“We directly fine-tune Molmo2 and Molmo2-ER with the same discrete-action architecture”

Near-zero action-expert output initialization used

“The final projection is initialized to produce near-zero velocity predictions at the beginning of post-training”

Autoregressive depth-token prefix conditioning optional

“MolmoAct2-Think keeps the same architecture and adds an autoregressive depth-token prefix before action prediction.”

2 more mentioned or evaluated
Hidden-state conditioning evaluated

multimodal architecture “We compare hidden-state conditioning, standard per-layer KV conditioning, and a per-head per-layer KV conditioning variant.”

Per-head per-layer key-value conditioning evaluated

multimodal architecture “In the per-head variant, the VLM keys and values remain separated by head, and each head is projected into the corresponding action-expert…”

filed at the root 3

Joint discrete and continuous action training core

“Overall, full-model adaptation with discrete and continuous action co-training provides the best average outcome.”

Multiple flow-time samples per action chunk used

“We use , so each robot sample contributes four points on the same flow trajectory while reusing the same visual-language context.”

VQ-VAE reconstruction loss used

“The VQVAE is trained with a standard reconstruction objective to minimize reconstruction loss between input RGB images and their…”

language modelling objective 3

Action-token-only cross-entropy loss core

“the loss is computed only on the action tokens”

Causal next-token prediction core

“adds robot-specific state, setup/control, and action-output tokens so that robot trajectories can be trained with the same next-token…”

Autoregressive discrete-action VLA pre-training core

“Pre-training first turns Molmo2-ER into an action-aware VLA using only autoregressive supervision”

distillation objective 1

Distillation of specialist depth estimates into autoregressively predicted depth tokens core

“the VLA is trained to predict this string autoregressively from the original RGB observation”

auxiliary loss 3

Knowledge insulation used

“The expert conditions on the VLM keys and values, but these tensors are detached before they enter the expert, so updates the action…”

Attention-mask isolation of packed examples used

“their text, visual tokens, state tokens, and action targets remain separated by the training attention mask, so the model does not…”

Depth-prefix token replacement noise during fine-tuning used

“10% of depth-code input tokens are replaced by uniformly sampled depth codes, while the prediction targets remain unchanged.”

filed at the root 1

Unfreezing tokenizer embeddings and language-model head used

“we follow the training scheme of LLaVA-AURORA (Bigverdi et al., 2025) by unfreezing the tokenizer embedding and lm head.”

training precision 2

BF16 mixed-precision training used

“We use automatic mixed precision with bfloat16 for most operations, while keeping numerically sensitive operations such as layer…”

FP32 layer normalization and rotary position embeddings used

“As an exception, we compute layer normalization and Rotary Position Embeddings (RoPE) in fp32.”

training parallelism 4

Fully Sharded Data Parallel used

“We train in PyTorch using Fully Sharded Data Parallel (FSDP)”

Cross-device average token-count loss normalization used

“each device divides by the average number of loss tokens across all devices rather than by its local number of loss tokens.”

Gradient-hook synchronization for replicated LoRA parameters used

“we register a gradient hook on those tensors to synchronize their gradients across ranks before the optimizer step.”

Replicated, unsharded LoRA parameters under FSDP used

“we do not shard LoRA adapter parameters under FSDP. Instead, each GPU keeps a full copy of the LoRA parameters”

data sourcing 2

Evenly spaced video-frame sampling used

“for each of the cameras in the trajectory that are not frozen, we sample 12 frames that are each equally temporally spaced”

Separate collection of atomic motion primitives used

“We isolated and separately collected data for each atomic motion—open, pick, flip, put, and close—to build a comprehensive set of motion…”

data filtering 10

Low-frame-rate video sampling with an eight-frame cap core

“Video examples are sampled at up to 2 FPS and capped to at most 8 frames.”

Filtering no-op actions and unsuccessful demonstrations used

“we trained on a modified dataset that filtered out no-op actions and unsuccessful demonstrations”

Excluding evaluation-style datasets used

“removal of eval-style datasets”

Four-stage quality filtering for community SO-100/101 robot data used

“we apply a four-stage filtering pipeline: (i) structural validity checks”

Idle-frame filtering used

“the provided idle-frame filter, which retains only contiguous non-idle action segments of at least one second.”

License and codebase eligibility filtering used

“license/codebase eligibility checks”

Normalize task strings into natural-language instructions used

“raw task strings are normalized into concise natural-language task descriptions.”

Structural validation of robot dataset samples used

“required schema fields, valid action/state tensors, no NaN/corrupt samples”

TOPReward dataset quality gate used

“we keep datasets whose mean TOPReward over the last 3 sampled episodes is above a threshold obtained by averaging the TOPReward over a…”

TOPReward-based dataset quality filtering used

“we use TOPReward (Chen et al., 2026) to filter out 438 low-quality dataset entries”

synthetic data 4

Demonstration collection with alternative language-conditioned motion paths used

“we collected half of the number of demonstrations in the regular way and the other half exploring alternative paths towards the same…”

Randomize requested instruction length during VLM relabeling used

“To increase the diversity of prompts, we randomly sample a number and ask the VLM to make this instruction roughly that many words.”

VLM-based point annotation for visual traces used

“In contrast, VLM-based point annotations (e.g., Molmo (Deitke et al., 2024), RoboPoint (Yuan et al., 2024)) yield accurate, non-degenerate…”

VLM-based robot demonstration language re-annotation used

“we use an open-source VLM (Qwen3.5-27B) to re-annotate the datasets.”

data mixture & curriculum 13

Mixed training on action-only and depth-and-action examples core

“while uniformly mixing action-only and depth-and-action examples.”

Pre-training on action reasoning, auxiliary robot, and multimodal web data core

“MolmoAct is pre-trained on a mixture of action reasoning data, auxiliary robot data, and multimodal web data.”

Weighted sampling by assigned mixture rates core

“data is sampled at the following rates: RT-1 (20%), BridgeData V2 (12.5%), BC-Z (7.5%)”

Co-training vision-language-action models with multimodal web data used

“We include a mixture of multimodal web data from Molmo’s Supervised fine-tuning stage”

Episode-level camera-order randomization used

“For multi-camera robot episodes, we randomize the input camera order at the episode level to prevent the model from relying on a fixed…”

Heterogeneous embodiment-specific action representations for tokenizer training used

“the tokenizer is trained on a heterogeneous mix of "dialects," including Absolute Joint positions and Delta End-Effector velocities.”

Joint multi-task training used

“All eight tasks are trained jointly in a multi-task setup, with demonstrations from every task and variation pooled into a single training…”

Joint refinement on embodied and general multimodal data used

“on a mixture that interleaves our embodied corpus with Molmo2’s original multimodal mid-training data”

Multi-task batch mixing used

“Batches mix examples from multiple tasks.”

Paired side-and-wrist camera-view examples used

“by pairing each side view with the wrist view”

Transparent tokenizer training mixture used

“MolmoAct2-FAST Tokenizer is distinguished by its use of a fully transparent training mix”

Uniform sampling of action, depth, and joint depth-and-action tasks used

“Within robot data, post-training uniformly samples three tasks: action prediction, depth prediction, and depth-and-action prediction.”

Within-batch mixing of training sources used

“During fine-tuning, dataset mixing is performed within each batch so that each update contains examples from multiple sources.”

sequence packing 5

Dynamic-programming selection for sequence packing used

“Once the pool is full, a dynamic-programming solver selects the subset that maximizes”

Fixed-cap action-chunk packing used

“we pad the packed action-chunk axis up to a small fixed cap of five chunks per packed sequence when possible.”

On-the-fly sequence packing with attention masking used

“We use on-the-fly packing to combine multiple short examples into one 4200-token sequence.”

Quantized sequence packing used

“In practice, we solve a quantized version of the packing problem by rounding token counts to the nearest multiple of 32.”

Robot-example packing with per-example VLM context used

“each action chunk is paired with the sub-example that produced it, and the expert attends only to that example’s VLM context.”

tokenization 9

Quantile normalization of robot actions core

“we normalize using dataset quantiles and discretize into 256 uniform-width bins between the first and ninety-ninth percentiles”

Robot-specific token interface core

“by adding robot-specific tokens to the backbone vocabulary.”

Similarity-preserving monotonic assignment of BPE symbols to action bins core

“We then assign them monotonically to the 256 bins so that adjacent bins map to adjacent symbols”

DCT-BPE action compression used

“While we utilize the core logic of frequency-domain compression (DCT-BPE)”

MolmoAct2-FAST Tokenizer used

“representing the trajectory with a frequency-domain transform, quantizing the resulting coefficients, and applying byte-pair encoding to…”

Normalization and 256-bin discretization of continuous robot states used

“Continuous robot states are normalized using dataset statistics and discretized into one of 256 state tokens per scalar dimension.”

Normalize, pad, and tokenize continuous robot actions used

“Continuous actions are normalized, padded to a maximum width of 32 dimensions, and encoded by MolmoAct2-FAST Tokenizer into discrete…”

Padding tokenizer and language-model head to a multiple of 512 used

“we first pad the tokenizer and lm head to its next multiple of 512, then replace the first 130 tokens with our depth perception tokens”

Selectable robot-output styles optional

“Robot examples are formatted with one of three output styles.”

filed at the root 4

post-training core

“Post-training then attaches the continuous action expert to this already action-aware backbone”

Three-stage VLA training pipeline core

“MolmoAct2 follows a three-stage training pipeline”

Post-training for multi-embodiment robot control used

“This checkpoint is the post-trained, multi-embodiment MolmoAct2 model.”

Specialize-then-rehearse training used

“we build on the released Molmo2 checkpoint with a two-stage specialize-then-rehearse recipe.”

supervised fine-tuning 10

Full-model fine-tuning core

“Overall, full-model adaptation with discrete and continuous action co-training provides the best average outcome.”

Embodiment-specific fine-tuning core

“fine-tuning adapts that model quickly to a specific embodiment, environment, and use case”

Depth-token noise during fine-tuning core

“The baseline uses both 10% depth-token noise and the learned per-layer depth gate”

Low-Rank Adaptation used

“We fine-tune MolmoAct-7B-D using Low-Rank Adaptation (LoRA)”

Embodied specialization stage used

“we fine-tune for K steps on the Molmo2-ER corpus augmented with Tulu-3 (Lambert et al., 2025) text-only data to preserve language competence”

Fine-tuning on target-embodiment demonstrations used

“deployment on robots outside this set ... requires effective fine-tuning on target-embodiment demonstrations”

Fine-tuning on the combined LIBERO training suites used

“The MolmoAct2-LIBERO checkpoint is fine-tuned on the full LIBERO training mixture, combining the Spatial, Object, Goal, and Long suites…”

Robot-only fine-tuning used

“each run is robot-only: we do not include the multimodal VLM mixture or a separate VLM dataloader.”

Separate single-task policies for bimanual YAM tasks used

“We trained MolmoAct2-Post as eight separate single-task policies”

Fine-tuning for a target robot embodiment or benchmark optional

“Use this checkpoint for further fine-tuning on a target robot embodiment or benchmark.”

mid-training & continual pretraining 2

Mid-training core

“we conduct a second stage of mid-training using high-quality action reasoning data closely aligned with our target domain of household…”

Mid-training on the MolmoAct Dataset used

“MolmoAct-7B-D outperforms its counterpart without mid-training by an average margin of 5.5% across the three tasks, demonstrating that…”

1 more mentioned or evaluated
Action-expert-only fine-tuning evaluated

supervised fine-tuning “Tuning only the action expert is the clearest failure mode, dropping the average to 93.05%.”

Not used 2

Flow-loss updates through action-expert conditioning not used

supervised fine-tuning “we do not use knowledge insulation during fine-tuning: gradients from the flow loss are allowed to update the VLM through the…”

Freeze added-token input embeddings during embodiment fine-tuning not used

supervised fine-tuning “we do not tune the added-token input embeddings.”

filed at the root 2

Pre-deployment robot-action safety checks optional

“Where possible, actions should be monitored through interpretable intermediate outputs (adaptive depth map), simulation rollouts, action…”

Reusable-cache optimization optional

“an optimized eager path using reusable-cache optimizations”

decoding strategy 2

Action-chunk execution used

“execute full chunks before redoing action reasoning”

Autoregressive discrete action-token inference optional

“MolmoAct2 also exposes a discrete action path, in which the VLM autoregressively decodes action tokens for the entire action chunk.”

reasoning control 1

Adaptive reasoning core

“new depth tokens are computed only for novel visual signals”

KV cache management 2

Cached depth-token replay for unchanged cells used

“Unchanged cells are replayed from the previous predicted depth buffer and consumed by the model as known depth-token inputs.”

Preallocated static KV cache used

“using a preallocated static KV cache to make the decode state stable across steps.”

inference scheduling 3

Caching invariant action-expert intermediates used

“We therefore cache reusable action-expert intermediates, including context-dependent cross-attention states and fixed position-dependent…”

Eager adaptive depth scheduling used

“We therefore keep the adaptive scheduler eager, including span-level replay for unchanged cells, while using a preallocated static KV…”

Static-shape compilation used

“We use torch.compile for throughput and keep the ViT and LLM shapes static through padding and fixed sequence budgets”

inference kernel 1

CUDA Graph capture and replay used

“We further capture the fixed-shape flow loop with CUDA Graphs, reducing Python and kernel-launch overhead during repeated action generation.”

context management 2

Adaptive depth-token prediction with cached unchanged cells core

“Instead of re-predicting every depth code at every step, MolmoAct2-Think reuses cached codes for static regions and autoregressively…”

Cosine similarity used

“each patch is compared to the corresponding patch in the previous frame by cosine similarity. A cell is marked updated when the similarity…”

agentic scaffolding 7

Visual trace steering core

“we allow the user to draw a visual reasoning trace directly on the camera image to indicate the desired end-effector path”

Interactive natural-language steering used

“open-ended natural language instructions provided interactively by participants”

Matching training and inference prefixes used

“During inference, we provide the same formatted user message, assistant prefix, and trigger as the generation prefix”

Prompt descriptors for embodiment and control semantics used

“Setup and control descriptors make embodiment and action semantics explicit in the prompt.”

Prompting Molmo to locate the robot gripper used

“For each observation frame, we prompt Molmo with the instruction "point to the robot gripper"”

Test-time action steering through language or visual traces optional

“a web interface that enables users to modify the language instruction or sketch five points on the image for visual trace steering at test…”

Trajectory editing optional

“The behaviors can be executed directly, or steered via trajectory editing.”

filed at the root 2

Best-performing checkpoint selection used

“for evaluation, we select the best-performing checkpoint, which occurs at 40K updates.”

Pre-marked object-location evaluation used

“In each efficiency fine-tuning task evaluation, we pre-marked the locations of all task objects in the scene to ensure that the evaluation…”

filed at the root 1

Long-horizon task decomposition used

“This decomposition allows the policy to learn individual skill components in isolation before composing them into more complex behaviors.”

unfiled 31

Action chunking core

“we apply action chunking (Zhao et al., 2023) during post-training, formatting action predictions in fixed-length chunks”

Action reasoning via intermediate perception and trajectory representations core

“MolmoAct frames visuomotor control as an autoregressive sequence modeling problem augmented with an action chain-of-thought (CoT)”

Fixed-horizon chunked action prediction and execution core

“MolmoAct2 predicts fixed-horizon action chunks (30 steps at 30 Hz for YAM/SO-100/101, 15 at 15 Hz for DROID, 10 at 10 Hz for LIBERO) and…”

Flow matching for continuous action generation core

“learns a flow-matching velocity field over continuous robot action chunks.”

Generate discrete depth tokens adaptively for dynamic scene regions core

“MolmoAct2-Think precedes this continuous action generation with an adaptive generation of discrete depth tokens only for dynamic scene…”

Joint autoregressive and flow-matching training objective core

“The post-training objective combines the autoregressive loss from pre-training with the flow-matching loss”

Low-Rank Adaptation fine-tuning for task and embodiment adaptation core

“We adapt MolmoAct to post-train on single or multi-tasks via parameter-efficient LoRA fine-tuning.”

Open-loop execution of action chunks before policy re-query core

“executes them open-loop before re-querying the policy.”

Per-layer key-value conditioning of a flow-matching action expert core

“Post-training then attaches a flow-matching action expert with per-layer KV conditioning on the autoregressive backbone”

Spatially grounded action reasoning core

“MolmoAct grounds every step of its reasoning chain directly in the scene”

Three-stage action reasoning pipeline core

“integrates perception, planning, and control through a structured reasoning pipeline”

Visual reasoning traces core

“By combining depth perception tokens, visual reasoning traces, and action prediction”

Continuous action-generation inference path default

“We therefore use the continuous path as the default deployment option.”

Overlapping image crops with border context default

“The grid squares are then moved closer together to introduce a fixed overlap margin (default 4 patches, approximately 56 pixels), which…”

1st–99th percentile normalization of continuous action dimensions used

“Continuous dimensions are normalized with 1–99 percentile statistics, which limits the effect of outliers while preserving the useful…”

Apply stochastic image augmentation before normalization and resizing used

“The image augmentation applies the stochastic transforms before image normalization and final resizing.”

Bound robot actions by safety limits used

“The model’s action space should be bounded by the training data, robot controller limits, and task-specific safety constraints, including…”

Collecting action-composition diversity by exploring task-completing motion variants used

“Practically, this means inducing the robot to explore motion variants while still completing tasks, so the model learns rich…”

Downscaling collected images to match OXE pre-training resolution used

“To match OXE during pre-training, we downscale our collected images from 640480 to 320240 px while preserving the original aspect ratio.”

Euler integration from Gaussian noise for action generation used

“At inference, we initialize the trajectory from Gaussian noise and integrate the learned velocity field for a fixed number of Euler steps”

Evaluate multiple flow-matching times per chunk while reusing VLM context used

“In post-training, we evaluate multiple sampled flow-matching times for each robot action chunk while reusing the same VLM context.”

Full action-chunk execution before replanning used

“we set action chunk size to for evaluation on each task suites and execute full chunks before replanning.”

Global-average loss-token normalization across devices used

“we divide the device’s total loss by the global average number of loss-tokens per example across all devices.”

Mask padded action dimensions in flow training used

“the dimension mask zeroes padded action dimensions in the noisy inputs, targets, and predictions before averaging the loss over valid…”

Mask padded action-horizon steps from the flow loss used

“The horizon mask removes padded time steps from the flow loss”

Operate robot systems with human supervision used

“operate the system only in a safely configured environment with human supervision.”

Replacing padding-token IDs with depth-perception tokens used

“we simply replace the first 130 padding tokens with the depth perception tokens”

Separate normalization treatment for gripper commands used

“Gripper commands are handled separately from this continuous normalization, since they are typically binary or narrow-range open/close…”

Uniform 256-bin discretization of proprioceptive state used

“each state value is uniformly discretized into one of 256 state tokens”

VQ-VAE depth-map quantization into discrete depth-token codes used

“The codebook is produced by a pre-trained specialist depth estimator”

Simulation rollouts for action validation optional

“actions should be monitored through interpretable intermediate outputs (adaptive depth map), simulation rollouts, action limits”

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

SourceKindPublisherStatus
huggingface.co/allenai/MolmoAct2model cardAi2read
arxiv.org/abs/2605.02881technical reportAi2read
arxiv.org/abs/2508.07917technical reportAi2read
↑