single mention · evaluation
LLM judge for document-quality annotation
An LLM judge annotates the quality of documents by scoring them under many quality prompts, such as educational value, chat-likeness, and benchmark usefulness.1,2 For each prompt the judge returns structured JSON enforced via constrained decoding, and the resulting judgement axes are then distilled into binary classification tasks for cheaper classifiers.3,4
Trade-off Applying the judge at scale is costly, so it is run on a sample and distilled into faster classifiers for the rest of the crawl.5,6
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
We first scored a fixed 1M-document English sample of our curated CC data with a judge, Qwen3-32B
— tech reportPrompts include educational value, chat-likeness, encyclopedia-likeness, content versus format properties, binary ‘quality’ judgements, and benchmark usefulness
— tech reportFor each prompt, the judge returns structured JSON, enforced via constrained decoding against a schema
— tech reportwe distil each scalar judgement axis into a binary task (top quartile versus bottom quartile, discarding the middle two) and train three families of classifiers
— tech reportit could be wasteful to apply an LLM-as-a-judge to tens of billions of documents
— tech reportT o score the rest of the crawl, we distil each scalar judgement axis into a binary task
— tech report
Also called judge.
Filed under evaluation › judge