Open Models Almanac
Methodsevaluationjudge

single mention · evaluation

LLM judge for document-quality annotation

An LLM judge annotates the quality of documents by scoring them under many quality prompts, such as educational value, chat-likeness, and benchmark usefulness.1,2 For each prompt the judge returns structured JSON enforced via constrained decoding, and the resulting judgement axes are then distilled into binary classification tasks for cheaper classifiers.3,4

Trade-off Applying the judge at scale is costly, so it is run on a sample and distilled into faster classifiers for the rest of the crawl.5,6

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. We first scored a fixed 1M-document English sample of our curated CC data with a judge, Qwen3-32B — tech report
  2. Prompts include educational value, chat-likeness, encyclopedia-likeness, content versus format properties, binary ‘quality’ judgements, and benchmark usefulness — tech report
  3. For each prompt, the judge returns structured JSON, enforced via constrained decoding against a schema — tech report
  4. we distil each scalar judgement axis into a binary task (top quartile versus bottom quartile, discarding the middle two) and train three families of classifiers — tech report
  5. it could be wasteful to apply an LLM-as-a-judge to tens of billions of documents — tech report
  6. T o score the rest of the crawl, we distil each scalar judgement axis into a binary task — tech report

Also called judge.

Filed under evaluation › judge