Open Models Almanac
Modelsπ0.5

Physical Intelligence

π0.5

What π0.5’s own documents say it is built from — every method with the sentence that describes it.

Parameters
3.6B
Attention
PaliGemma-3B backbone, flow-matching action expert; co-training on web and robot data
Input
text+image -> action
Licence
Apache-2.0 (openpi code); Gemma terms (PaliGemma weights)
Released
2025-09-09

Against the consensus recipe

Its documents state 1 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

filed at the root 3

Flow-based vision-language-action model core

“We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM)”

Unified model for high-level and low-level inference core

“uses a unified architecture where the same model performs both high-level and low-level inference”

Downsized action expert used

“To speed up inference (which requires multiple forward passes of the action expert), we downsize the action expert to {width=1024…”

token mixer 4

Unidirectional VLM-to-action-expert information flow used

“In effect, information flows unidirectionally from the VLM to the action expert”

softmax attention 3

Bidirectional attention core

“Within each block, there is full bidirectional attention”

Three-block causal attention mask core

“uses a blockwise causal attention mask with 3 blocks”

Prefix-conditioned autoregressive action-token attention used

“FAST action tokens attend to this prefix and auto-regressively on previous action tokens”

channel mixer 1

mixture of experts 1

Action expert core

“implemented as a single transformer with two sets of weights (also known as experts [45]), where each token is routed to one of the experts”

normalization & residual 1

Adaptive RMSNorm used

“applies adaptive RMSNorm to inject the timestep information to each layer of the action expert”

prediction head 1

Flow-matching action head default

“This model currently supports only the flow-matching action head for π₀.₅ training and inference.”

multimodal architecture 5

Separate flow-matching action expert core

“we adapt the model to also have an action expert, as with π0, in order to both represent actions with finer granularity and enable more…”

Routing image and language tokens to the VLM backbone core

“The images and language prompt, , are routed to the larger VLM backbone, which we initialize from PaliGemma.”

Routing robotics-specific inputs to an action expert core

“The inputs not seen during VLM pre-training, , are routed to the action expert.”

Masking missing robot image slots used

“For robots with fewer than three images, we also mask out the missing image slots.”

Separate robotics-specific token weights used

“using a separate set of weights for the robotics-specific (action and state) tokens led to an improvement in performance.”

3 more mentioned or evaluated
Diffusion decoding for vision-language-action policies evaluated

prediction head “we compare the performance of both models on the tasks from Section VI-B”

Regression-head action prediction mentioned

prediction head “A number of works have also proposed alternatives to tokenization, for example by using regression heads”

Semantic action representations using language sub-tasks mentioned

“One line of work uses “semantic” action representations like language sub-tasks [21, 2, 4], or keypoints [50, 32, 25, 19].”

language modelling objective 3

Autoregressive discrete-action prediction core

“the π₀-FAST model, an autoregressive VLA, based on the FAST action tokenizer”

Discrete-tokenized pretraining used

“with discrete tokenized training in the pre-training phase”

High-level subtask prediction used

“we manually annotate all data with semantic descriptions of the subtasks and train to jointly predict the subtask labels (as text) as well…”

auxiliary loss 2

Knowledge insulation used

“an upgraded version of π₀ with better open-world generalization trained with knowledge insulation”

Attention masking to prevent information leakage between action representations used

“do not attend to FAST action tokens to avoid information leakage between the two representations of actions”

1 more mentioned or evaluated
Implicit high-level inference from training data evaluated

language modelling objective “no high-level inference at runtime but includes high-level data in training, which may teach the model about subtasks implicitly”

filed at the root 1

Exponential moving average of model weights used

“compute an EMA of the network weights with weight 0.999.”

optimizer 1

AdamW used

“We use the AdamW optimizer [45] ... without weight decay”

learning-rate schedule 1

Linear learning-rate warmup followed by a constant rate used

“We train all policies using a short linear learning rate warm-up (1k steps) and then a constant learning rate of 5e-5.”

training stability 2

Global-norm gradient clipping used

“clip gradient magnitude to 1”

Exponential moving average (EMA) optional

“you may want to consider disabling EMA”

training parallelism 2

Fully Sharded Data Parallel optional

“use `--fsdp-devices <n>` where `<n>` is your number of GPUs, to enable fully-sharded data parallelism, which reduces memory usage in…”

Model parallelism optional

“you can also use multiple GPUs with model parallelism to reduce per-GPU memory requirements by configuring `fsdp_devices`”

data sourcing 1

Web-data training used

“performance drops substantially when cross-embodiment data (ME or CE) or web data (WD) is removed”

data filtering 5

Filtering idle all-zero-action timesteps used

“filter out any idle timesteps with all-zero actions during training”

Idle-data filtering used

“We have added an improved idle filter for DROID training.”

Per-dataset action-dimension quantile normalization used

“All action data is normalized to using the 1% and 99% quantile of each action dimension of the individual dataset.”

Random image crop, resize, rotation, and color-jitter augmentation used

“We apply image augmentation (random crop, resizing, rotation, and color jittering) to all input images”

Success-only episode filtering used

“we train only on the episodes marked as “success” (75k episodes)”

synthetic data 1

Expert-provided language demonstrations of high-level subtasks used

“expert users providing “language demonstrations,” selecting appropriate sub-task commands to command the robot to perform mobile…”

data mixture & curriculum 12

Heterogeneous-data co-training core

“It is also co-trained jointly with data from the web, as well as high-level prediction data for outputting language commands based on…”

Pre-training on diverse robot-manipulation data core

“consists of pre-training on highly diverse data, followed by either out-of-box evaluation or fine-tuning to complex downstream tasks”

Cross-embodiment robot training used

“cross-embodiment transfer and diverse data co-training are critical for generalization across a range of tasks”

Co-training on robot demonstrations and multimodal data used

“co-trained on robot demonstrations and large-scale multimodal data”

Cross-embodied robot data mixture used

“we train the -FAST model from the previous section on the cross-embodied robot data mixture”

Down-weighting overrepresented task–robot combinations used

“over-represented combinations are down-weighted.”

Large-scale multi-source robot-manipulation pre-training mixture used

“Our pre-training mixture consists of 10,000 hours of dexterous manipulation data from 7 different robot configurations and 68 tasks, in…”

Multimodal web-data co-training used

“we include a diverse set of web data involving image captioning (CapsFusion [87], COCO [12]), question answering (Cambrian-7M [77], PixMo…”

Random camera-view selection during DROID training used

“Since DROID provides two external camera views per episode, we randomly sample the third-person view during training.”

Random selection among language annotations during DROID training used

“DROID provides three natural language annotations for each training episode, and we randomize over them during training.”

Training on subtask-prediction data used

“a significant portion of that benefit is already obtained simply by including subtask prediction data in the training mixture”

Universal action tokenizer with multiple action-space parameterizations used

“we include versions with multiple action space parametrizations: joint space, end-effector world frame, and end-effector camera frame, to…”

tokenization 9

Byte-pair encoding (BPE) for action-token compression core

“train a byte pair encoding (BPE) tokenizer to losslessly compress it into dense action tokens”

Discrete cosine transform (DCT) action compression core

“we use a compression algorithm based on the discrete cosine transform (DCT)”

FAST action tokenizer core

“based on the FAST action tokenizer”

FAST+ core

“We therefore base our method off of the discrete cosine transform (DCT) encoding”

Scale-and-round quantization of DCT coefficients core

“which we implement through a scale-and-round operation”

Quantile normalization of robot actions used

“We first normalize the input actions, such that the 1st and 99th quantile of values in the training dataset for each action dimension maps…”

256-bin proprioceptive-state discretization used

“for the proprioceptive state, we apply a bin tokenization pre-processing, akin to RT-2’s action tokenization [10], discretizing into 256…”

Column-first DCT-coefficient flattening used

“We choose the former, since we find that predicting the low-frequency components”

Padding actions to 32 dimensions used

“Before tokenization, all actions are padded to 32 dimensions to accommodate action spaces of different dimensionality.”

1 more mentioned or evaluated
FAST action-token training evaluated

tokenization “training with FAST tokens is more effective in terms of compute than pure diffusion based training”

Not used 1

Per-dimension 256-bin robot-action discretization not used

tokenization “Prior robotic policies of this sort typically use naïve tokenization strategies based on a per-dimension, per-timestep binning scheme”

filed at the root 3

Pre-training followed by post-training for robot policies core

“training then proceeds in two stages: a pre-training stage intended to adapt the model to diverse robotic tasks, and a post-training stage…”

post-training used

“uses a hybrid training procedure, with discrete tokenized training in the pre-training phase, and training with a flow matching action…”

Task-specific post-training used

“the goal of the post-training phase is to provide the model with the ability to skillfully and fluently execute the desired downstream task.”

supervised fine-tuning 6

Full-model fine-tuning used

“We fine-tune the VLA models for robot action prediction, without weight freezing.”

Adaptation of vision-language models using robot data used

“indicating the importance of adapting VLMs with robot data”

Fine-tuning on downstream robot tasks used

“followed by either out-of-box evaluation or fine-tuning to complex downstream tasks”

Fine-tuning with high-quality data used

“complex tasks like laundry following require fine-tuning with high-quality data”

Low-Rank Adaptation optional

“Fine-Tuning (LoRA)”

Fine-tuning optional

“Intended use: Base model to fine tune on your specific use case”

mid-training & continual pretraining 1

Broad robot-data pretraining used

“The goal of the pre-training phase is to expose the model to a diverse range of tasks”

1 more mentioned or evaluated
Training a downstream policy only on high-quality task data evaluated

supervised fine-tuning “Training on only this high-quality data results in a brittle model”

decoding strategy 2

Greedy decoding default

“During inference, we use simple greedy autoregressive decoding”

Temperature sampling for bimanual tasks optional

“except for the bi-manual robot tasks ... where we found a small temperature of to be helpful”

reasoning control 2

Hierarchical subtask inference core

“The high-level inference mechanism in takes in a high-level command (e.g., “clean the bedroom”) and outputs the subtask to complete (e.g.…”

High-level language policy for task decomposition used

“We use such a high-level policy to assist our model with high-level strategy for several of our experimental tasks”

KV cache management 2

Robot-state KV caching across sampling steps core

“preventing it from attending to the final block allows its corresponding keys and values to be cached during sampling”

Prefix key-value caching during action inference used

“inference can be implemented efficiently by caching the attention keys and values for the prefix”

1 more mentioned or evaluated
GPT-4 high-level policy evaluated

agentic scaffolding “use GPT-4 as the high-level policy, evaluating the importance of training the high-level policy on robot data”

filed at the root 1

Task-specific progress scoring rubric used

“For each task, we design a score rubric that measures progress on the task, and use this for our quantitative results.”

human & real-world evaluation 2

Interleaved policy evaluation used

“Evaluations were carried out by interleaving execution of policies to control for environmental changes.”

Zero-shot evaluation in unseen environments used

“we test the policy in a completely unseen environment, with a new table setup, background, novel objects, viewpoint and table height”

1 more mentioned or evaluated
Pretraining and fine-tuning ablation comparison evaluated

“we use these tasks to compare to ablations of our approach, evaluating after pre-training and fine-tuning, out of the box after…”

unfiled 24

Flow matching core

“Training objective: flow matching”

Conditional flow matching for continuous action distributions core

“uses conditional flow matching [28, 32] to model the continuous distribution of actions.”

Long-horizon planning core

“indicating the importance of structured subtask prediction and long-horizon planning”

Structured subtask prediction core

“indicating the importance of structured subtask prediction and long-horizon planning”

Low-timestep-biased flow-matching timestep sampling used

“we designed a timestep sampling distribution that emphasizes low timesteps (high noise levels)”

Action chunking for robot control used

“we use an action chunking architecture [57] with flow matching”

Action expert used throughout training used

“while always uses the action expert”

Exclude flow-matching timesteps above a threshold used

“timesteps above a given threshold are not sampled at all”

Explicit high-level inference for structured subtask prediction and planning used

“explicit high-level inference improves performance across tasks”

Forward Euler integration for action generation used

“We use the forward Euler integration rule:”

Full-prefix attention masking used

“A full prefix mask is used on images, prompt tokens, and proprioceptive state”

Open-loop execution of predicted action chunks used

“we opted not to aggregate actions and instead execute action chunks open-loop”

PD tracking controllers for commanded robot targets used

“These targets are tracked with simple PD controllers, without any additional trajectory planning or collision detection.”

Ten-step flow-matching action-chunk generation used

“both perform 10 steps of flow matching to predict the action chunk”

Threshold-based exclusion of high flow-matching timesteps used

“Timesteps above the threshold are excluded from sampling”

Zero-padding action vectors to a shared fixed dimensionality used

“For robots with lower-dimensional configuration and action spaces, we zero-pad the action vectors.”

Zero-padding robot configuration and action vectors used

“For robots with lower-dimensional configuration and action spaces, we zero-pad the configuration and action vectors.”

Reuse pretrained state and action normalization statistics during fine-tuning optional

“functionality for reloading normalization statistics for state / action normalization from pre-training”

5 more mentioned or evaluated
Expert-human high-level policy evaluated

“use an expert human as an “oracle” high-level policy, to provide an upper bound on performance”

Finite Scalar Quantization (FSQ) action-tokenization baseline evaluated

“we can consider an alternative compression scheme that instead trains a model to produce a quantized representation of the action chunk…”

Vector-quantized action representation evaluated

“Such approaches train a vector-quantized encoder-decoder network”

Keypoint-based action representations mentioned

“or keypoints [50, 32, 25, 19]”

Logit-normal flow-matching timestep sampling mentioned

“Esser et al. [14] instead propose sampling from a logit-normal distribution that emphasizes the middle timesteps”

Not used 1

Temporal ensembling of actions from separate inference calls not used

“We tried temporal ensembling [57] early on and found that it hurt policy performance”

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

SourceKindPublisherStatus
huggingface.co/lerobot/pi05_basemodel cardHugging Face (LeRobot port)read
arxiv.org/abs/2504.16054technical reportPhysical Intelligenceread
arxiv.org/abs/2410.24164technical reportPhysical Intelligenceread
arxiv.org/abs/2501.09747technical reportPhysical Intelligenceread
github.com/Physical-Intelligence/openpicode repoPhysical Intelligenceread
↑