Open Models Almanac
Methodsevaluationbenchmark

single mention · evaluation

PaperBench hierarchical rubric evaluation

PaperBench is a benchmark that evaluates AI agents' ability to replicate AI research papers, including understanding contributions, developing a codebase, and executing experiments.1 It grades replication hierarchically: rubrics decompose each task into smaller sub-tasks with clear grading criteria, yielding many individually gradable tasks.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. PaperBench [35] evaluates the ability of AI agents to replicate state-of-the-art AI research — gpt-oss-120b & gpt-oss-20b Model Card
  2. we develop rubrics that hierarchically decompose each replication task into smaller sub-tasks with clear grading criteria — gpt-oss-120b & gpt-oss-20b Model Card

Also called PaperBench.

Filed under evaluation › benchmark