single mention · evaluation
RoboLab
RoboLab is a simulated robot benchmark of 120 tabletop tasks, each phrased three ways from vague to specific.1 It targets models trained on real-world robot data so they cannot memorize the test environment, and new tasks can be generated in minutes to keep it from going stale.2
Trade-off Its rankings correlate strongly with real-robot rankings on RoboArena, though that correlation rests on just four models.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
Its 120 tabletop tasks, such as stacking blocks in a given order or putting all the green fruit on a plate, are each phrased three ways, from vague to specific
— Black Forest Labs’ Flux 3 Action Applies Video Expertise to a World-Action ModelIt’s designed for models trained on real-world robot data, so they can’t memorize the test environment
— Black Forest Labs’ Flux 3 Action Applies Video Expertise to a World-Action Modelits rankings correlate strongly with those of RoboArena
— Black Forest Labs’ Flux 3 Action Applies Video Expertise to a World-Action Model
Also called RoboLab simulated robot benchmark.
Filed under evaluation › benchmark