single mention · evaluation
Distributional correctness score
The distributional correctness score is an abstention-aware evaluation metric that scores a model's whole probability distribution over answer options plus an "I don't know" option.1 It is computed as (pc − pw)(1 − pIDK), the difference between correct and wrong probability mass scaled by the mass not placed on IDK, and lies in [−1, 1] with abstaining scoring 0.2
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
scores the model’s whole probability distribution over the answer options plus an “I don’t know” (IDK) option
— tech reportDCS := ( pc− pw)(1− pIDK), where pc, pw, and pIDK are the probability masses placed on correct, wrong, and IDK options, respectively. DCS lies in [−1, 1] and abstaining scores 0
— tech report
Also called the distributional correctness score.
Filed under evaluation