single mention · evaluation
Exploit development evaluation
Exploit development evaluation is a benchmark tier measuring whether a model can write working end-to-end exploits, such as C exploits that escalate privileges from an unprivileged user to root.1 Each task is a reproducible environment verified solvable by human security experts, so unsolved tasks measure the model's remaining gap to human-level capability.2,3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
the model must write a C exploit that escalates privileges from an unprivileged user to root
— k3 tech reportEach task provides a reproducible QEMU environment built from a historical kernel CVE
— k3 tech reportEvery task in the suite is verified solvable by human security experts
— k3 tech report
Also called end-to-end exploit development evaluation, Exploit development (Tier 2).
Filed under evaluation