Robbyant (Ant Group)
LingBot-VLA 2.0
What LingBot-VLA 2.0’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 6.4B
- Attention
- 6B; ~20K h dual-arm data across 9 embodiments (v1)
- Input
- text+image -> action
- Licence
- Apache-2.0
- Released
- 2026-07-07
Against the consensus recipe
Its documents state 1 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictionnot in its documents
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modenot in its documents
- Multi-Token Predictionnot in its documents
- FP8not in its documents
filed at the root 1
“These components are organized via a Mixture-of-Transformers (MoT) architecture”
token mixer 2
softmax attention 2
“we implement blockwise causal attention for modeling the joint sequence”
“vision-language and action modalities are processed through distinct transformer pathways, coupled by a shared self-attention mechanism”
channel mixer 8
mixture of experts 8
“we introduce a token-level loss-free MoE architecture for large-scale multi-embodiment VLA pretraining.”
expert routing 1
“To avoid the strong competition among experts induced by softmax normalization, we apply a sigmoid-activated function to the router logits”
expert load balancing 3
“we adopt an auxiliary-loss-free strategy inspired by DeepSeek-V3 [19] to promote load balancing within the MoE architecture.”
“each expert maintains a routing correction bias.”
“At each load-balancing update, the correction bias is adjusted according to the sign of the deviation from the mean expert load:”
fine-grained experts 2
“Fine-grained expert segmentation and shared expert isolation allow universal priors and specialized embodiment/task patterns to coexist…”
“we adopt fine-grained expert segmentation and shared expert isolation, in which a lightweight shared expert preserves universal priors…”
positional encoding 2
“a shared learnable 2D spatial positional embedding for both RGB and depth tokens; and (2) a modality embedding that distinguishes the…”
“At each scale, UV positional encodings—derived from a circular mapping of image coordinates—are injected to preserve spatial layout and…”
prediction head 1
“we first convert the depth into a point cloud and then predict an dexterous hand pose using a diffusion policy”
multimodal architecture 7
“we adopt a convolutional decoder architecture, termed ConvStack, which is more appropriate for dense geometric reconstruction”
“The decoder follows a pyramid structure composed of a shared convolutional neck and multiple task-specific heads.”
“we apply the learnable queries corresponding to three-view operational images”
“LingBot-VLA-V2 appends current and future perceptual queries to the visual/text tokens.”
“We apply separate patch embedding layers to the two input modalities: the 3-channel RGB image and the single-channel depth map.”
“The policy is conditioned on RGB features extracted by DINOv2 (ViT-L/14) and point cloud features from a Point Transformer”
“depth injection”
filed at the root 4
“Predictive dynamics modeling: future prediction is used as a proxy task”
“we adopt a vision distillation approach inspired by recent works”
“The predicted depth map is supervised using an L1 loss applied directly to the ground-truth depth map. Only pixels with valid depth values…”
“We train DINO-Video on 5M video clips spanning Internet, egocentric, and robotic data, using video-adapted DINO and iBOT self-distillation…”
distillation objective 4
“the current and future queries, , are trained to predict the corresponding depth representations from the current and future manipulation…”
“We align the VLM learnable queries and LingBot-Depth tokens by minimizing the distillation loss”
“These queries are distilled from LingBot-Depth and DINO-Video, encouraging causal inference to capture both current scene geometry and…”
“depth-free version and a depth-distilled version.”
auxiliary loss 1
“interpretation of missing regions (“holes”) in raw depth maps as “natural masks””
optimizer 1
“we employ a differential learning rate strategy: parameters in the encoder backbone are optimized with a base learning rate of , while all…”
learning-rate schedule 1
“After warm-up, a step decay scheduler reduces both learning rates by a factor of 0.5 every 25,000 iterations.”
training precision 2
“mixed-precision training using BF16 is enabled throughout to improve computational efficiency and reduce memory consumption”
“performing reductions in torch.float32 to ensure numerical stability, while utilizing torch.bfloat16 for storage and communication”
training stability 1
“Gradient clipping with a maximum norm of 1.0 is applied to stabilize optimization”
training parallelism 2
“We employ Fully Sharded Data Parallel (FSDP)—a highly efficient PyTorch implementation of the Zero Redundancy Optimizer (ZeRO)—to shard…”
“we construct specific “shard groups” exclusively for the action expert modules”
filed at the root 1
“Define feature mapping (states / actions / images) from raw keys to unified feature space”
data sourcing 2
“20,000 hours of real-world data from 9 popular dual-arm robot configurations.”
“a redesigned data pipeline curates around 60,000 hours of pre-training data, including 50,000 hours of robot trajectories across 20 robot…”
data filtering 14
“Calculate normalization statistics over your dataset”
“Human annotators are employed to identify discrepancies between the projected robot and the videos”
“a depth patch (token) whose values are entirely missing is always masked”
“using a 20% valid-frame ratio as the minimum threshold.”
“If this duration accounts for more than 95% of the entire episode, the episode is also discarded.”
“For patches containing a mixture of valid and invalid depth values, we assign a higher probability (e.g., 0.75 in our training) of being…”
“we randomly sample additional fully valid depth tokens to complete the masking set”
“we employ a redesigned data processing pipeline to filter noisy samples, yielding 50,000 hours of high-quality robotic data.”
“An episode is discarded if either the jerk or any derivative Z-score exceeds a predefined threshold.”
“We then apply trajectory-level quality control to the reconstructed and standardized data.”
“We first apply a unified video-level VLM pre-filter to all candidate videos, regardless of source.”
“an algorithmic screening procedure automatically excludes episodes exhibiting technical anomalies”
“human reviewers validate the filtered dataset using synchronized multi-view video streams”
“we do the left-right check and filter out the inconsistent pixel values from the depth maps”
synthetic data 6
“We generate these annotations with a fully automated pipeline built on a vision–language model”
“we employ Qwen3.6-27B [25] to segment each manipulation video into a sequence of temporally contiguous subtasks and to generate the…”
“we artificially corrupt depth maps with Gaussian noise and apply our masking strategy during training”
“we follow FoundationStereo [33] to compute the stereo disparity of the left-right IR pairs to obtain the pseudo depth labels”
“we explicitly simulate the imaging process of real-world active RGB-D cameras to generate realistic depth observations with natural…”
“Our data curation pipeline consists of two parallel streams: a synthetic pipeline built on self-hosted 3D assets”
policy distillation 1
“we adopt a dual-query distillation framework inspired by recent works [35], where LingBot-Depth and our robotics-aware DINO-Video serve as…”
inference kernel 2
“we leverage FlexAttention to optimize computation”
“we apply operator fusion (via torch.compile) to reduce kernel launch overhead”
filed at the root 1
“Support open-loop evaluation.”
human & real-world evaluation 2
“All models are tested sequentially on the same hardware-task pair in randomized order”
“Each model undergoes 15 trials per task-robot pair for statistical robustness”
1 more mentioned or evaluated
“we corrupt the remaining depth values with additive Gaussian noise and shot-noise-like perturbations inspired by Kinect noise models”
other 4
filed at the root 4
“we use a 55-dimensional canonical vector representation for both states and actions.”
“the unified representation supports arms, end-effectors, grippers, dexterous hands, waist, head, and mobile-base signals instead of only…”
“mainly resort to the Bundle Adjustment (BA) on the tracked VO points to track the camera motion”
“The rendered stereo images are then processed using the widely adopted semi-global matching (SGM) algorithm to produce sensor-like depth…”
unfiled 20
“the future query is trained to predict the feature of the future frame at horizon .”
“our architecture provides the full, unmasked RGB image as a condition”
“the training objective is to characterize the conditional distribution through conditional flow matching”
“The transition between these tasks is governed simply by the masking strategy within the Transformer.”
“we introduce Masked Depth Modeling (MDM) to achieve dense, pixel-aligned scene geometry”
“we run egocentric SLAM to estimate camera intrinsics and per-frame camera extrinsics”
“Baidu Cloud's optimization solutions notably reduced our GPU memory consumption by 29.2% during model training.”
“apply hand pose estimation to recover MANO parameters in the camera coordinate frame.”
“Data augmentation includes random resized cropping and horizontal flipping, along with a set of synthetic image degradations—specifically…”
“we initialize the framewise extrinsics using SE-3 identity”
“we build a scalable RGB-D capture prototype”
“we use the online version of SpatialTrackerV2 [34] that does not depend on the VGGT [27]-Frontend for initial pose and depth estimation as…”
“object poses (positions and orientations) are randomized within the workspace for each trajectory to prevent overfitting to specific…”
“human hand-object interactions are retargeted to the dexterous hand configuration via 3D keypoint correspondence”
“Enable Torch Compile for inference.”
4 more mentioned or evaluated
“We generate incomplete depth maps by randomly masking out spatial regions of varying sizes (blocks) from the ground-truth depth”
“For the loss function, L2 achieves the best average performance, improving over L1”
“By preserving these larger corrective motions, MeanStd achieves the best average success rate of .”
“relative joint actions substantially outperform absolute joint actions”
Not used 1
“There is no global BA applied to enable the efficiency.”
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| huggingface.co/robbyant/lingbot-vla-v2-6b | model card | Robbyant | read |
| arxiv.org/abs/2607.06403 | technical report | Robbyant | read |
| arxiv.org/abs/2601.18692 | technical report | Robbyant | read |
| arxiv.org/abs/2601.17895 | technical report | Robbyant | read |
| github.com/Robbyant/lingbot-vla | code repo | Robbyant | read |