Open Models Almanac
Methodsdata curationdata mixture & curriculum

general family · filed under data curation

Heterogeneous-data co-training

Heterogeneous-data co-training structures training corpora as a data pyramid, organizing heterogeneous sources by scale from web data to real-world robot execution.1 This strategy learns across all data layers by inferring pseudo-actions for action-less videos using a latent-action codebook and a trained inverse dynamics model.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we structure the VLA training corpora as a data pyramid, illustrated in Fig. 1. Rather than treating the training datasets as a homogeneous pool, we organize heterogeneous sources by scale: large quantities of web data and human videos lay the base of the pyramid; synthetic data generated with physics simulations and/or augmented by off-the-shelf neural models form the middle layer, and real-world data collected on the physical robot hardware complete the top — GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
  2. We develop an effective co-training strategy to learn across the entire data pyramid in both pre- and post-training phases. To train our model with action-less data sources, such as human videos and neural-generated videos, we learn a latent-action codebook (Ye et al., 2025) and also use a trained inverse dynamics model (IDM) to infer pseudo-actions — GR00T N1: An Open Foundation Model for Generalist Humanoid Robots

Also called co-training strategy, Co-training on diverse data sources, diverse data co-training, co-training recipe, co-training framework.

sources
2
models
2
labs adopt it
2
strongest
core

Documented in

Physical Intelligence

Evidence

4 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 2core 2

π0.5

Physical Intelligenceadopted1 source

It is also co-trained jointly with data from the web, as well as high-level prediction data for outputting language commands based on robot observations.

coreunclearin π0.5
2 more from this model

we propose to include a range of different data sources to create the model

coreunclearin π0.5

cross-embodiment transfer and diverse data co-training are critical for generalization across a range of tasks

useddata curationin π0.5
↑