Open Models Almanac
Methods

single mention · not yet filed

Latent-action representation learned from video

Latent-action representation learned from video generates latent actions by training a VQ-VAE model to extract features from consecutive image frames from videos.1 The encoder takes the current frame and the future frame of a video with a fixed window size and outputs the latent action.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we instead generate latent actions by training a VQ-VAE model to extract features from consecutive image frames from videos — GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
  2. The encoder takes the current frame and the future frame of a video with a fixed window size and outputs the latent action — GR00T N1: An Open Foundation Model for Generalist Humanoid Robots

Also called latent actions.