Open Models Almanac
Methodsevaluationbenchmark

single mention · evaluation

Capture-the-Flag (CTF) evaluation

A Capture-the-Flag (CTF) evaluation tests a model on cybersecurity challenges graded by the skill level needed to solve them.1 The model runs in a headless Linux environment with offensive security tools preinstalled and a harness for executing commands, with performance reported over multiple rollouts.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. divided into the level of proficiency needed to solve the challenge, ultimately organized by whether the challenge required high school, collegiate, or professional-level cybersecurity knowledge — gpt-oss-120b & gpt-oss-20b Model Card
  2. we give it access to a headlessLinux distribution with common offensive cybersecurity tools preinstalled as well as a harness which allows the model to call those tools — gpt-oss-120b & gpt-oss-20b Model Card

Also called CTF challenges.

Filed under evaluation › benchmark