SourcesPhysical Intelligence
publisher · 4 sources
Physical Intelligence
4 sources read from this publisher, newest first, with the methods each one describes.
π0.5\pi_{0.5}: a Vision-Language-Action Model with Open-World Generalization
research paper · technical report · 2025-04-22 · 37 methods
Flow matchingBidirectional attentionHeterogeneous-data co-trainingpost-trainingAutoregressive discrete-action predictionPre-training followed by post-training for robot policiesSeparate flow-matching action expertCross-embodiment robot trainingHierarchical subtask inferenceLow-timestep-biased flow-matching timestep samplingUnified model for high-level and low-level inferenceAction expert used throughout trainingAdaptation of vision-language models using robot dataAdaptive RMSNormAttention masking to prevent information leakage between action representationsDiscrete-tokenized pretrainingExpert-human high-level policyExpert-provided language demonstrations of high-level subtasksExplicit high-level inference for structured subtask prediction and planningFAST action-token trainingFull-prefix attention maskingGPT-4 high-level policyHigh-level subtask predictionImplicit high-level inference from training dataInterleaved policy evaluationLong-horizon planningMultimodal web-data co-trainingPD tracking controllers for commanded robot targetsPer-dataset action-dimension quantile normalizationPrefix-conditioned autoregressive action-token attentionRandom image crop, resize, rotation, and color-jitter augmentationStructured subtask predictionThreshold-based exclusion of high flow-matching timestepsTraining on subtask-prediction dataUnidirectional VLM-to-action-expert information flowWeb-data trainingZero-padding action vectors to a shared fixed dimensionality
FAST: Efficient Action Tokenization for Vision-Language-Action Models Thanks: ∗: Core contributors Correspondence to:…
research paper · technical report · 2025-01-16 · 29 methods
AdamWGlobal-norm gradient clippingFAST+Greedy decodingFull-model fine-tuningLinear learning-rate warmup followed by a constant rateExponential moving average of model weightsPer-dimension 256-bin robot-action discretizationQuantile normalization of robot actionsRandom camera-view selection during DROID trainingRandom selection among language annotations during DROID trainingSuccess-only episode filtering256-bin proprioceptive-state discretizationByte-pair encoding (BPE) for action-token compressionColumn-first DCT-coefficient flatteningCross-embodied robot data mixtureDiffusion decoding for vision-language-action policiesDiscrete cosine transform (DCT) action compressionFiltering idle all-zero-action timestepsFinite Scalar Quantization (FSQ) action-tokenization baselineKeypoint-based action representationsPadding actions to 32 dimensionsRegression-head action predictionScale-and-round quantization of DCT coefficientsSemantic action representations using language sub-tasksTemperature sampling for bimanual tasksUniversal action tokenizer with multiple action-space parameterizationsVector-quantized action representationZero-shot evaluation in unseen environments
π0\pi_{0}: A Vision-Language-Action Flow Model for General Robot Control Thanks: Physical Intelligence, San Francisco, California, USA.…
research paper · technical report · 2024-10-31 · 35 methods
Bidirectional attentionFlow-based vision-language-action modelPre-training followed by post-training for robot policiesSeparate flow-matching action expertAction expertCross-embodiment robot trainingLow-timestep-biased flow-matching timestep samplingTask-specific post-trainingAction chunking for robot controlBroad robot-data pretrainingConditional flow matching for continuous action distributionsDown-weighting overrepresented task–robot combinationsDownsized action expertExclude flow-matching timesteps above a thresholdFine-tuning on downstream robot tasksFine-tuning with high-quality dataForward Euler integration for action generationHigh-level language policy for task decompositionLarge-scale multi-source robot-manipulation pre-training mixtureLogit-normal flow-matching timestep samplingMasking missing robot image slotsOpen-loop execution of predicted action chunksPre-training on diverse robot-manipulation dataPrefix key-value caching during action inferencePretraining and fine-tuning ablation comparisonRobot-state KV caching across sampling stepsRouting image and language tokens to the VLM backboneRouting robotics-specific inputs to an action expertSeparate robotics-specific token weightsTask-specific progress scoring rubricTemporal ensembling of actions from separate inference callsTen-step flow-matching action-chunk generationThree-block causal attention maskTraining a downstream policy only on high-quality task dataZero-padding robot configuration and action vectors
openpi
software documentation · code repo · 11 methods
Low-Rank AdaptationFully Sharded Data ParallelKnowledge insulationFlow-based vision-language-action modelAutoregressive discrete-action predictionFlow-matching action headExponential moving average (EMA)FAST action tokenizerIdle-data filteringModel parallelismReuse pretrained state and action normalization statistics during fine-tuning