Physical Intelligence
π0.5
What π0.5’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 3.6B
- Attention
- PaliGemma-3B backbone, flow-matching action expert; co-training on web and robot data
- Input
- text+image -> action
- Licence
- Apache-2.0 (openpi code); Gemma terms (PaliGemma weights)
- Released
- 2025-09-09
Against the consensus recipe
Its documents state 1 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationused
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertsnot in its documents
- Multi-Token Predictionnot in its documents
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modenot in its documents
- Multi-Token Predictionnot in its documents
- FP8not in its documents
filed at the root 3
“We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM)”
“uses a unified architecture where the same model performs both high-level and low-level inference”
“To speed up inference (which requires multiple forward passes of the action expert), we downsize the action expert to {width=1024…”
token mixer 4
“In effect, information flows unidirectionally from the VLM to the action expert”
softmax attention 3
“Within each block, there is full bidirectional attention”
“uses a blockwise causal attention mask with 3 blocks”
“FAST action tokens attend to this prefix and auto-regressively on previous action tokens”
channel mixer 1
mixture of experts 1
“implemented as a single transformer with two sets of weights (also known as experts [45]), where each token is routed to one of the experts”
normalization & residual 1
“applies adaptive RMSNorm to inject the timestep information to each layer of the action expert”
prediction head 1
“This model currently supports only the flow-matching action head for π₀.₅ training and inference.”
multimodal architecture 5
“we adapt the model to also have an action expert, as with π0, in order to both represent actions with finer granularity and enable more…”
“The images and language prompt, , are routed to the larger VLM backbone, which we initialize from PaliGemma.”
“The inputs not seen during VLM pre-training, , are routed to the action expert.”
“For robots with fewer than three images, we also mask out the missing image slots.”
“using a separate set of weights for the robotics-specific (action and state) tokens led to an improvement in performance.”
3 more mentioned or evaluated
prediction head “we compare the performance of both models on the tasks from Section VI-B”
prediction head “A number of works have also proposed alternatives to tokenization, for example by using regression heads”
“One line of work uses “semantic” action representations like language sub-tasks [21, 2, 4], or keypoints [50, 32, 25, 19].”
language modelling objective 3
“the π₀-FAST model, an autoregressive VLA, based on the FAST action tokenizer”
“with discrete tokenized training in the pre-training phase”
“we manually annotate all data with semantic descriptions of the subtasks and train to jointly predict the subtask labels (as text) as well…”
auxiliary loss 2
“an upgraded version of π₀ with better open-world generalization trained with knowledge insulation”
“do not attend to FAST action tokens to avoid information leakage between the two representations of actions”
1 more mentioned or evaluated
language modelling objective “no high-level inference at runtime but includes high-level data in training, which may teach the model about subtasks implicitly”
filed at the root 1
“compute an EMA of the network weights with weight 0.999.”
learning-rate schedule 1
“We train all policies using a short linear learning rate warm-up (1k steps) and then a constant learning rate of 5e-5.”
training stability 2
“clip gradient magnitude to 1”
“you may want to consider disabling EMA”
training parallelism 2
“use `--fsdp-devices <n>` where `<n>` is your number of GPUs, to enable fully-sharded data parallelism, which reduces memory usage in…”
“you can also use multiple GPUs with model parallelism to reduce per-GPU memory requirements by configuring `fsdp_devices`”
data sourcing 1
“performance drops substantially when cross-embodiment data (ME or CE) or web data (WD) is removed”
data filtering 5
“filter out any idle timesteps with all-zero actions during training”
“We have added an improved idle filter for DROID training.”
“All action data is normalized to using the 1% and 99% quantile of each action dimension of the individual dataset.”
“We apply image augmentation (random crop, resizing, rotation, and color jittering) to all input images”
“we train only on the episodes marked as “success” (75k episodes)”
synthetic data 1
“expert users providing “language demonstrations,” selecting appropriate sub-task commands to command the robot to perform mobile…”
data mixture & curriculum 12
“It is also co-trained jointly with data from the web, as well as high-level prediction data for outputting language commands based on…”
“consists of pre-training on highly diverse data, followed by either out-of-box evaluation or fine-tuning to complex downstream tasks”
“cross-embodiment transfer and diverse data co-training are critical for generalization across a range of tasks”
“co-trained on robot demonstrations and large-scale multimodal data”
“we train the -FAST model from the previous section on the cross-embodied robot data mixture”
“over-represented combinations are down-weighted.”
“Our pre-training mixture consists of 10,000 hours of dexterous manipulation data from 7 different robot configurations and 68 tasks, in…”
“we include a diverse set of web data involving image captioning (CapsFusion [87], COCO [12]), question answering (Cambrian-7M [77], PixMo…”
“Since DROID provides two external camera views per episode, we randomly sample the third-person view during training.”
“DROID provides three natural language annotations for each training episode, and we randomize over them during training.”
“a significant portion of that benefit is already obtained simply by including subtask prediction data in the training mixture”
“we include versions with multiple action space parametrizations: joint space, end-effector world frame, and end-effector camera frame, to…”
tokenization 9
“train a byte pair encoding (BPE) tokenizer to losslessly compress it into dense action tokens”
“we use a compression algorithm based on the discrete cosine transform (DCT)”
“based on the FAST action tokenizer”
“We therefore base our method off of the discrete cosine transform (DCT) encoding”
“which we implement through a scale-and-round operation”
“We first normalize the input actions, such that the 1st and 99th quantile of values in the training dataset for each action dimension maps…”
“for the proprioceptive state, we apply a bin tokenization pre-processing, akin to RT-2’s action tokenization [10], discretizing into 256…”
“We choose the former, since we find that predicting the low-frequency components”
“Before tokenization, all actions are padded to 32 dimensions to accommodate action spaces of different dimensionality.”
1 more mentioned or evaluated
tokenization “training with FAST tokens is more effective in terms of compute than pure diffusion based training”
Not used 1
tokenization “Prior robotic policies of this sort typically use naïve tokenization strategies based on a per-dimension, per-timestep binning scheme”
filed at the root 3
“training then proceeds in two stages: a pre-training stage intended to adapt the model to diverse robotic tasks, and a post-training stage…”
“uses a hybrid training procedure, with discrete tokenized training in the pre-training phase, and training with a flow matching action…”
“the goal of the post-training phase is to provide the model with the ability to skillfully and fluently execute the desired downstream task.”
supervised fine-tuning 6
“We fine-tune the VLA models for robot action prediction, without weight freezing.”
“indicating the importance of adapting VLMs with robot data”
“followed by either out-of-box evaluation or fine-tuning to complex downstream tasks”
“complex tasks like laundry following require fine-tuning with high-quality data”
“Fine-Tuning (LoRA)”
“Intended use: Base model to fine tune on your specific use case”
mid-training & continual pretraining 1
“The goal of the pre-training phase is to expose the model to a diverse range of tasks”
1 more mentioned or evaluated
supervised fine-tuning “Training on only this high-quality data results in a brittle model”
decoding strategy 2
“During inference, we use simple greedy autoregressive decoding”
“except for the bi-manual robot tasks ... where we found a small temperature of to be helpful”
reasoning control 2
“The high-level inference mechanism in takes in a high-level command (e.g., “clean the bedroom”) and outputs the subtask to complete (e.g.…”
“We use such a high-level policy to assist our model with high-level strategy for several of our experimental tasks”
KV cache management 2
“preventing it from attending to the final block allows its corresponding keys and values to be cached during sampling”
“inference can be implemented efficiently by caching the attention keys and values for the prefix”
1 more mentioned or evaluated
agentic scaffolding “use GPT-4 as the high-level policy, evaluating the importance of training the high-level policy on robot data”
filed at the root 1
“For each task, we design a score rubric that measures progress on the task, and use this for our quantitative results.”
human & real-world evaluation 2
“Evaluations were carried out by interleaving execution of policies to control for environmental changes.”
“we test the policy in a completely unseen environment, with a new table setup, background, novel objects, viewpoint and table height”
1 more mentioned or evaluated
“we use these tasks to compare to ablations of our approach, evaluating after pre-training and fine-tuning, out of the box after…”
unfiled 24
“Training objective: flow matching”
“uses conditional flow matching [28, 32] to model the continuous distribution of actions.”
“indicating the importance of structured subtask prediction and long-horizon planning”
“indicating the importance of structured subtask prediction and long-horizon planning”
“we designed a timestep sampling distribution that emphasizes low timesteps (high noise levels)”
“we use an action chunking architecture [57] with flow matching”
“while always uses the action expert”
“timesteps above a given threshold are not sampled at all”
“explicit high-level inference improves performance across tasks”
“We use the forward Euler integration rule:”
“A full prefix mask is used on images, prompt tokens, and proprioceptive state”
“we opted not to aggregate actions and instead execute action chunks open-loop”
“These targets are tracked with simple PD controllers, without any additional trajectory planning or collision detection.”
“both perform 10 steps of flow matching to predict the action chunk”
“Timesteps above the threshold are excluded from sampling”
“For robots with lower-dimensional configuration and action spaces, we zero-pad the action vectors.”
“For robots with lower-dimensional configuration and action spaces, we zero-pad the configuration and action vectors.”
“functionality for reloading normalization statistics for state / action normalization from pre-training”
5 more mentioned or evaluated
“use an expert human as an “oracle” high-level policy, to provide an upper bound on performance”
“we can consider an alternative compression scheme that instead trains a model to produce a quantized representation of the action chunk…”
“Such approaches train a vector-quantized encoder-decoder network”
“or keypoints [50, 32, 25, 19]”
“Esser et al. [14] instead propose sampling from a logit-normal distribution that emphasizes the middle timesteps”
Not used 1
“We tried temporal ensembling [57] early on and found that it hurt policy performance”
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| huggingface.co/lerobot/pi05_base | model card | Hugging Face (LeRobot port) | read |
| arxiv.org/abs/2504.16054 | technical report | Physical Intelligence | read |
| arxiv.org/abs/2410.24164 | technical report | Physical Intelligence | read |
| arxiv.org/abs/2501.09747 | technical report | Physical Intelligence | read |
| github.com/Physical-Intelligence/openpi | code repo | Physical Intelligence | read |