Open Models Almanac
Methodsevaluationbenchmark

single mention · evaluation

SWE-bench Verified evaluation

SWE-bench Verified is a benchmark that evaluates AI models' ability to solve real-world software issues.1 It uses a human-validated subset of SWE-bench that fixes grading errors, under-specified problem statements, and overly specific unit tests, so models are graded accurately.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. evaluates AI models’ ability to solve real-world software issues — gpt-oss-120b & gpt-oss-20b Model Card
  2. the human-validated subset of SWE-bench that more reliably evaluates — gpt-oss-120b & gpt-oss-20b Model Card

Also called SWE-bench Verified.

Filed under evaluation › benchmark