SourcesAi2
publisher · 3 sources
Ai2
3 sources read from this publisher, newest first, with the methods each one describes.
MolmoAct2 Action Reasoning Models for Real-World Deployment
research paper · technical report · 2026-05-04 · 108 methods
Low-Rank AdaptationCUDA Graph capture and replayRotary Position EmbeddingBF16 mixed-precision trainingKnowledge insulationPer-layer key-value conditioningpost-trainingEmbodiment-specific fine-tuningEpisode-level camera-order randomizationFlow-matching action expertFull-model fine-tuningMultiple flow-time samples per action chunkAdaptive reasoningCausal next-token predictionLearned per-layer depth gateMasking action tokens from expert conditioningMolmoAct2-FAST TokenizerOn-the-fly sequence packing with attention maskingScaled Dot-Product Attention (SDPA)Single resized crop per image1st–99th percentile normalization of continuous action dimensionsAction-expert-only fine-tuningAdaptive depth-token prediction with cached unchanged cellsApply stochastic image augmentation before normalization and resizingAttention-based pooling of local visual windowsAttention-mask isolation of packed examplesAutoregressive depth-token prefix conditioningAutoregressive discrete action-token inferenceAutoregressive discrete-action VLA pre-trainingBest-performing checkpoint selectionBidirectional self-attention with VLM cross-attentionCached depth-token replay for unchanged cellsCaching invariant action-expert intermediatesContinuous action-generation inference pathContinuous DiT-style action expertCosine similarityCross-device average token-count loss normalizationDCT-BPE action compressionDepth-conditioning gate initializationDepth-prefix token replacement noise during fine-tuning
allenai/MolmoAct2 · Hugging Face
first party release · model card · 2026-05-04 · 8 methods
Per-layer key-value conditioningFlow-matching action expertPre-deployment robot-action safety checksBound robot actions by safety limitsFine-tuning for a target robot embodiment or benchmarkOperate robot systems with human supervisionPost-training for multi-embodiment robot controlSimulation rollouts for action validation
MolmoAct Action Reasoning Models that can Reason in Space
research paper · technical report · 2025-08-11 · 54 methods
Low-Rank AdaptationFully Sharded Data ParallelMid-trainingAction chunkingBF16 mixed-precision trainingAction-token-only cross-entropy lossAction-chunk executionDepth-aware perception tokensFiltering no-op actions and unsuccessful demonstrationsQuantile normalization of robot actionsScaled Dot-Product Attention (SDPA)Visual trace steeringAction reasoning via intermediate perception and trajectory representationsCo-training vision-language-action models with multimodal web dataCollecting action-composition diversity by exploring task-completing motion variantsDemonstration collection with alternative language-conditioned motion pathsDistillation of specialist depth estimates into autoregressively predicted depth tokensDownscaling collected images to match OXE pre-training resolutionFP32 layer normalization and rotary position embeddingsFull action-chunk execution before replanningGlobal-average loss-token normalization across devicesGradient-hook synchronization for replicated LoRA parametersIndex-token marking for multi-image inputsInteractive natural-language steeringLearned image-padding-status embeddingsLong-horizon task decompositionLow-Rank Adaptation fine-tuning for task and embodiment adaptationMid-training on the MolmoAct DatasetMulti-layer ViT feature concatenationMulti-scale image croppingMulti-task batch mixingOverlapping image crops with border contextPadding tokenizer and language-model head to a multiple of 512Paired side-and-wrist camera-view examplesPre-marked object-location evaluationPre-training on action reasoning, auxiliary robot, and multimodal web dataPredicting end-effector trajectory traces as intermediate visual plansPrompting Molmo to locate the robot gripperReplacing padding-token IDs with depth-perception tokensReplicated, unsharded LoRA parameters under FSDP