Open Models Almanac
Methodsoptimizationtraining parallelism

specific method · filed under optimization

Fully Sharded Data Parallel

Fully Sharded Data Parallel is a highly efficient PyTorch implementation of the Zero Redundancy Optimizer (ZeRO).1 It shards optimizer states, model parameters, and gradients across devices to minimize the memory footprint.1

Trade-off It reduces memory usage in exchange for slower training.2

How this definition was written

Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.

  1. We employ Fully Sharded Data Parallel (FSDP)—a highly efficient PyTorch implementation of the Zero Redundancy Optimizer (ZeRO)—to shard optimizer states, model parameters, and gradients, thereby minimizing memory footprint — A Pragmatic VLA Foundation Model
  2. reduces memory usage in exchange for slower training — openpi

Also called FSDP, fully-sharded data parallelism, Fully Sharded Data Parallel training, Fully Sharded Data Parallel (FSDP), fully sharded data parallelism (FSDP), FSDP with optional CPU offloading.

sources
5
models
5
labs adopt it
5
strongest
used

Documented in

Allen Institute for AI
Robbyant (Ant Group)
OpenVLA project (Stanford et al.)
Physical Intelligence
Toyota Research Institute

Evidence

6 quotes from 5 sources, one card per model, each opening with its strongest.

How sources treat it
optional 4used 2

LingBot-VLA 2.0

Robbyant (Ant Group)adopted1 source

We employ Fully Sharded Data Parallel (FSDP)—a highly efficient PyTorch implementation of the Zero Redundancy Optimizer (ZeRO)—to shard optimizer states, model parameters, and gradients

usedunclearin LingBot-VLA training codebaseRobbyant

VLA Foundry (Foundry-VLA-1.7B)

Toyota Research Instituteadopted1 source

The training loop supports the standard levers for scaling distributed training – FSDP (with optional CPU offloading)

optionaloptimizationin VLA FoundryToyota Research Institute
1 more from this model

The loop composes cleanly with the distributed training primitives users expect: FSDP with optional CPU offloading

optionaloptimizationin VLA FoundryToyota Research Institute

π0.5

Physical Intelligenceadopted1 source

use `--fsdp-devices <n>` where `<n>` is your number of GPUs, to enable fully-sharded data parallelism, which reduces memory usage in exchange for slower training

optionaloptimizationin openpi trainingPhysical Intelligence
↑