Open Models Almanac
Methodsevaluation

single mention · evaluation

Exploit development evaluation

Exploit development evaluation is a benchmark tier measuring whether a model can write working end-to-end exploits, such as C exploits that escalate privileges from an unprivileged user to root.1 Each task is a reproducible environment verified solvable by human security experts, so unsolved tasks measure the model's remaining gap to human-level capability.2,3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. the model must write a C exploit that escalates privileges from an unprivileged user to root — k3 tech report
  2. Each task provides a reproducible QEMU environment built from a historical kernel CVE — k3 tech report
  3. Every task in the suite is verified solvable by human security experts — k3 tech report

Also called end-to-end exploit development evaluation, Exploit development (Tier 2).

Filed under evaluation