Open Models Almanac
Methodsevaluationbenchmark

single mention · evaluation

RoboLab

RoboLab is a simulated robot benchmark of 120 tabletop tasks, each phrased three ways from vague to specific.1 It targets models trained on real-world robot data so they cannot memorize the test environment, and new tasks can be generated in minutes to keep it from going stale.2

Trade-off Its rankings correlate strongly with real-robot rankings on RoboArena, though that correlation rests on just four models.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. Its 120 tabletop tasks, such as stacking blocks in a given order or putting all the green fruit on a plate, are each phrased three ways, from vague to specific — Black Forest Labs’ Flux 3 Action Applies Video Expertise to a World-Action Model
  2. It’s designed for models trained on real-world robot data, so they can’t memorize the test environment — Black Forest Labs’ Flux 3 Action Applies Video Expertise to a World-Action Model
  3. its rankings correlate strongly with those of RoboArena — Black Forest Labs’ Flux 3 Action Applies Video Expertise to a World-Action Model

Also called RoboLab simulated robot benchmark.

Filed under evaluation › benchmark