Open Models Almanac
Methodsevaluationjudge

single source · evaluation

Agent-as-a-Judge

Agent-as-a-Judge is an evaluation method in which an autonomous agent judges whether generated code meets requirements.1 It deploys the generated project, then the judge agent interacts with the live UI and inspects code and terminal output to render pass/fail verdicts per check-item.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we employ a GUI agent to simulate human testing behavior to interactively verify each check item and assign scores based on the fulfillment of requirements — GLM-5: from Vibe Coding to Agentic Engineering
  2. each generated project is deployed in a Docker container and built to verify static correctness. Successfully built instances are then handed to an autonomous Judge Agent — GLM-5: from Vibe Coding to Agentic Engineering

Also called Agent-as-a-Judge evaluation method.

Filed under evaluation › judge