LingBot-VLA 2.0
Robbyant (Ant Group)We employ Fully Sharded Data Parallel (FSDP)—a highly efficient PyTorch implementation of the Zero Redundancy Optimizer (ZeRO)—to shard optimizer states, model parameters, and gradients
specific method · filed under optimization
Fully Sharded Data Parallel is a highly efficient PyTorch implementation of the Zero Redundancy Optimizer (ZeRO).1 It shards optimizer states, model parameters, and gradients across devices to minimize the memory footprint.1
Trade-off It reduces memory usage in exchange for slower training.2
Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.
We employ Fully Sharded Data Parallel (FSDP)—a highly efficient PyTorch implementation of the Zero Redundancy Optimizer (ZeRO)—to shard optimizer states, model parameters, and gradients, thereby minimizing memory footprint— A Pragmatic VLA Foundation Model
reduces memory usage in exchange for slower training— openpi
Also called FSDP, fully-sharded data parallelism, Fully Sharded Data Parallel training, Fully Sharded Data Parallel (FSDP), fully sharded data parallelism (FSDP), FSDP with optional CPU offloading.
6 quotes from 5 sources, one card per model, each opening with its strongest.
We employ Fully Sharded Data Parallel (FSDP)—a highly efficient PyTorch implementation of the Zero Redundancy Optimizer (ZeRO)—to shard optimizer states, model parameters, and gradients
We train in PyTorch using Fully Sharded Data Parallel (FSDP)
The training loop supports the standard levers for scaling distributed training – FSDP (with optional CPU offloading)
The loop composes cleanly with the distributed training primitives users expect: FSDP with optional CPU offloading
use `--fsdp-devices <n>` where `<n>` is your number of GPUs, to enable fully-sharded data parallelism, which reduces memory usage in exchange for slower training
fully sharded data parallelism (FSDP, Zhao et al. [77])