Hugging Face
SmolVLA
What SmolVLA’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 0.45B
- Attention
- SmolVLM2-500M backbone, flow-matching action expert
- Input
- text+image -> action
- Licence
- Apache-2.0
- Released
- 2025-06-03
Against the consensus recipe
Its documents state 0 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertsnot in its documents
- Multi-Token Predictionnot in its documents
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningevaluated
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modenot in its documents
- Multi-Token Predictionnot in its documents
- FP8not in its documents
filed at the root 1
“SmolVLA is a lightweight VLA composed of a compact pretrained VLM, and an action expert trained with flow matching.”
token mixer 4
“Cross-attention (CA), where action tokens attend to the VLM’s features”
softmax attention 2
“Self-attention (SA), where action tokens attend to each other (causally—only to the past)”
“We employ a causal attention mask for the SA layers, ensuring that each action token can only attend to past tokens within the chunk”
hybrid layer stacking 1
“we employ an interleaved approach, where each block contains either a CA or a SA layer.”
channel mixer 1
mixture of experts 1
“SmolVLA’s action expert is a compact transformer (~100M parameters) that generates action chunks”
multimodal architecture 7
“reduce its hidden size to 75% of the VLM’s, keeping the model lightweight for deployment.”
“Leveraging smaller pretrained VLMs”
“Using fewer visual tokens”
“we use features from intermediate layers.”
“We use SmolVLM2 as our VLM backbone.”
“SmolVLA uses only the global image at runtime to keep inference lightweight and fast.”
“the action expert only attends to VLM features up to a configurable layer NN during training, set to half the total layers.”
3 more mentioned or evaluated
softmax attention “bidirectional interaction harms performance.”
token mixer “no interaction between action tokens (pure CA)”
multimodal architecture “feeding the sensorimotor states to the VLM (by projecting them into token space)”
filed at the root 1
“- -policy.gradient_checkpointing=true”
training precision 1
“we leverage bfloat16 precision and torch.compile()”
data sourcing 2
“It is pretrained on a curated mix of publicly available, community-contributed datasets designed to reflect real-world variation.”
“matching the performance of much larger VLAs through community-driven data”
data filtering 5
“we manually mapped each camera to a standardized view type—prioritizing top, wrist, and side perspectives–and renamed them as OBS_IMAGE_1…”
“we selected a subset of 481 community datasets obtained from Hugging Face, filtered according to embodiment type, episode count, overall…”
“observations are compared in joint-space, and near-duplicates are dropped.”
“After a meticulous manual review (special thanks to Marina Barannikov), we curated a collection of 487 high-quality datasets”
“We used a customfiltering tool created by Alexandre Chapin and Ville Kuosmanen to select datasets based on frame count, visual quality…”
synthetic data 3
“To improve quality and standardize the textual input across datasets, we used Qwen2.5-VL-3B-Instruct to generate concise, action-oriented…”
“For each dataset, we sampled representative frames and provided them alongside the original instruction.”
“the model was prompted to rewrite the instruction in under 30 characters, starting with an action verb”
data mixture & curriculum 1
“Unless specified otherwise, SmolVLA is always trained in a multi-task setting.”
sequence packing 1
“we maintain a fixed sequence length and batch size, discarding any excess frames in an episode that do not fit a complete batch.”
filed at the root 2
“SmolVLA goes through a pretraining phase on general manipulation data, followed by task-specific post-training.”
“SmolVLA goes through a pretraining phase on general manipulation data, followed by task-specific post-training.”
supervised fine-tuning 4
“Base model to fine tune on your specific use case”
“Our current approach primarily relies on imitation learning.”
“Multitask finetuning further boosts performance, showing strong task transfer capabilities even in low-data regimes.”
“We train only the action expert module, keeping the VLM frozen.”
1 more mentioned or evaluated
rollout & RL infrastructure “the async mode decouples action execution and policy inference, allowing predictions and control to run in parallel.”
filed at the root 1
“Overlapping actions from successive chunks are stitched with a simple merge rule to avoid jitter.”
inference scheduling 2
“we propose an asynchronous inference stack that enables faster adaptation and responsiveness in real-world manipulation tasks.”
“When the queue length falls below a threshold (e.g., 70%), we send an observation to a Policy Server”
1 more mentioned or evaluated
decoding strategy “the robot can execute multiple actions from the predicted chunk before processing new observations”
unfiled 21
“Training objective: flow matching”
“Our model predicts chunks of actions”
“It is trained using a flow matching objective, which teaches the model to guide noisy samples back to the ground truth.”
“during training, we add random noise to the robot’s real action sequences and ask the model to predict the “correction vector””
“Skipping half of the layers of the vision model for faster inference and smaller size”
“SmolVLA limits the number of visual tokens to 64 per frame during both training and inference.”
“chunk sizes between 10 and 50 provide a good balance between the robot reactivity and efficiency.”
“unused views were dropped during training.”
“triggering chunk prediction while the control loop is still consuming a previously available queue, aggregating it with the newly incoming…”
“In simulation, we perform inference by sampling new observations and predicting a new action after each executed action.”
“in addition to a pixel shuffle operation, limiting the visual tokens to 64 per frame.”
“using PixelShuffle as an efficient shuffling technique.”
“torch.compile() that JIT-compiles PyTorch code into optimized kernels.”
“consuming actions from a readily available queue until a threshold condition on the number of remaining actions in the queue is met.”
“For real-world evaluation, we perform synchronous inference: the model samples new observations only after executing the full chunk of…”
“Key design choices include: (i) skipping layers in the VLM”
5 more mentioned or evaluated
“updating observations more frequently significantly improves success rate”
“a standard regression L1 loss on the predicted versus ground-truth action chunks.”
“passing them directly to the action expert.”
“we also test a variant sampling every second VLM layer (Skip % 2)”
“using only the first half of the VLM layers gives a good trade-off between performance and compute.”
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| huggingface.co/lerobot/smolvla_base | model card | Hugging Face | read |
| arxiv.org/abs/2506.01844 | technical report | Hugging Face | read |
| huggingface.co/blog/smolvla | official blog | Hugging Face | read |