Toyota Research Institute
VLA Foundry (Foundry-VLA-1.7B)
What VLA Foundry (Foundry-VLA-1.7B)’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 1.7B
- Attention
- LLM -> VLM -> VLA trained from scratch in one framework; diffusion/flow-matching policy; Qwen3-VL variant (2.1B)
- Input
- text+image -> action
- Licence
- Apache-2.0
- Released
- 2026-04-17
Against the consensus recipe
Its documents state 0 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertsnot in its documents
- Multi-Token Predictionnot in its documents
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modenot in its documents
- Multi-Token Predictionnot in its documents
- FP8not in its documents
prediction head 2
“Foundry-VLM-1.3B vision-language backbone + (condition on last 4 layers) flow-matching diffusion action head (24 layers, 1024 hidden dim…”
“while additional heads such as a diffusion policy are already implemented in the codebase”
multimodal architecture 2
“(condition on last 4 layers)”
“we also train a VLA with the pretrained Qwen3-VL 2B model Bai et al., 2025 as backbone.”
1 more mentioned or evaluated
prediction head “integrating further variants – for example, autoregressive discrete action tokenizations – requires only a new head module”
optimization 11
filed at the root 3
“It expresses training budgets as a number of samples instead of a number of steps, so that runs at different batch sizes or GPU counts…”
“The training loop supports the standard levers for scaling distributed training – FSDP (with optional CPU offloading), mixed precision…”
“The training loop supports the standard levers for scaling distributed training – FSDP (with optional CPU offloading), mixed precision…”
training precision 1
“The training loop supports the standard levers for scaling distributed training – FSDP (with optional CPU offloading), mixed precision…”
training stability 1
“Actions and proprioceptive states are normalized during dataloading time and denormalized at inference time.”
training parallelism 2
“Built on FSDP2 with WebDataset streaming.”
“The training loop supports the standard levers for scaling distributed training – FSDP (with optional CPU offloading)”
training runtime 3
“dataloader state checkpointing for exact restarts”
“We rely on deterministic per-rank RNG seeding, dataloader state checkpointing for exact restarts”
“The training loop is deliberately model-agnostic. Unpacking a batch into inputs and targets, masking the loss, and reducing model outputs…”
1 more mentioned or evaluated
“torch.compile, and exponential moving average of weights”
filed at the root 2
“During dataset preprocessing, each training sample is constructed around an anchor timestep within an episode.”
“Robotics raw data can come from any source (simulation logs, real-robot recordings, etc.), as long as a converter knows how to read it and…”
data sourcing 2
“VLA Foundry has custom scripts to convert raw datasets to the WebDataset tar shards described above.”
“Training data: 102M samples from simulated and real-world bimanual manipulation tasks (with resample)”
data filtering 2
“the fields listed in proprioception_fields are each extracted, normalized, and concatenated along the feature dimension”
“To avoid degenerate samples with excessive padding, samples whose required padding exceeds configurable thresholds (max_padding_left…”
data mixture & curriculum 7
“Specify dataset sources and ratios at dataloader time for easy dataset mixing and batch balancing across modalities.”
“Flexible multi-modal training with probabilistic dataset mixing and dataloaders that support text, image-caption, and robotics data”
“each dataset contributes its own shards, statistics, and modalities with weighted dataset proportions”
“VLA Foundry natively supports dataset mixing with command-line arguments.”
“Unless otherwise noted, Foundry-VLA-1.7B and Foundry-Qwen3VLA-2.1B-MT are trained on a multi-task mixture of both real and simulation data”
“Trained on both simulated and real-world manipulation data.”
“the --data.dataset_weighting parameter, which handles the batch balancing ratios; a 1:2:1 weighting corresponds to 25%/50%/25% of each…”
sequence packing 3
“Raw data is converted to WebDataset Breuel, 2020 tar shards through a per-modality preprocessing stage.”
“For robotics datasets, this is done in multiple stages. First, it creates a frames folder in the output directory, where each sample is…”
“At episode boundaries, sequences are padded using a configurable padding strategy (copy, zero, or reflect).”
mid-training & continual pretraining 1
“the first trained fully from scratch through our LLM→VLM→VLA pipeline”
1 more mentioned or evaluated
supervised fine-tuning “finetuning the multi-task model on single seen tasks further improves performance in aggregate.”
context management 1
“Including past timesteps allows the model to condition on recent action history and proprioceptive context; predicting multiple future…”
filed at the root 2
“At runtime, create_model() looks up the registry by model.type and dispatches to the correct factory.”
“The steps of the WebDataset pipeline are defined sequentially, and notably support both WebDataset built-in functions (e.g.…”
training framework 2
“Training in VLA Foundry is done in a single training loop that is shared across every model stage in the pipeline”
“VLA Foundry supports both from-scratch training and pretrained backbones from Hugging Face.”
infrastructure service 1
“Preprocessing runs in parallel with Ray Moritz et al., 2018”
other 5
filed at the root 5
“Rotations are represented in the 6D continuous rotation format Zhou et al., 2019 throughout, with conversion to and from matrices…”
“with conversion to and from matrices performed via Gram–Schmidt orthogonalization.”
“VLA Foundry supports both absolute and relative action representations, which are stored as separate fields in the dataset.”
“Relative actions are computed with respect to the robot’s actual pose at the anchor timestep, i.e., the frame at which a prediction is made.”
“supports four normalization methods: standard deviation, min-max, and two percentile-based variants (percentile_1_99 and percentile_5_95).”
unfiled 23
“Actions are chunked Zhao et al., 2023 in a configurable window of past and future time steps around an anchor: the future portion…”
“proprioception uses only the past and current timesteps within the window (i.e., indices ), whereas actions span the full past-and-future…”
“Proprioceptive observations are causally restricted to past and current time steps.”
“Presets themselves are composable – a YAML file can inherit from others, so experiments are expressed by combining building blocks rather…”
“Each model is registered via a @register_model decorator on its factory function”
“Self-registering models and batch handlers via decorators.”
“The output action sequence is trained with the flow-matching objective Lipman et al., 2022.”
“immutable frozen dataclasses that prevent hidden configuration changes at runtime”
“our RoboticsNormalizer supports global and per-timestep schemes, including percentile-based variants”
“Configurations follow a three-level precedence: command-line arguments override YAML preset files, which override dataclass defaults.”
“Each model selects a BatchHandler – typically one of the shared handlers defined per modality or model type (LLM, VLM, and DP-VLA) – which…”
“a shared training stack with end-to-end control, from language pretraining to action-expert fine-tuning”
“VLA Foundry instead provides a shared training stack with end-to-end control, from language pretraining to action-expert fine-tuning.”
“we balance the per-task sample size for each policy to ensure that the aggregate represents an unbiased estimate of the policy’s…”
“We use ray Moritz et al., 2018 to parallelize data preprocessing.”
“each dataset retains a serialized t-digest sketch Dunning and Ertl, 2019 during preprocessing, and the sketches are merged at training…”
“The training loop supports the standard levers for scaling distributed training – FSDP (with optional CPU offloading), mixed precision…”
“For percentile estimation and merging, we use t-digest Dunning and Ertl, 2019.”
“Proprioceptive state is specified via a separate --proprioception_fields parameter, distinct from --action_fields.”
“Since datasets are processed individually with their own per-dataset stats.json files, we support merging multiple stats.json files…”
“In per-timestep scope, each timestep within the action window has its own mean and scale derived from statistics computed at that relative…”
“Entirely new training paradigms can register an additional handler via @register_batch_handler.”
1 more mentioned or evaluated
“gradient checkpointing, torch.compile, and exponential moving average of weights”
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| huggingface.co/TRI-ML/Foundry-VLA-1.7B-full | model card | TRI | read |
| arxiv.org/abs/2604.19728 | technical report | TRI | read |
| tri-ml.github.io/vla_foundry | official blog | TRI | read |