Open Models Almanac
Methodsevaluationbenchmark

methods · evaluation

benchmark

33 methods filed at this node or below it, from the sources of 12 models.

In this branch 33

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 33

all mentioned once
Living in-house benchmark suite core · 1 source · 1 quote
BrowseComp with Search used · 1 source · 1 quote
Capture-the-Flag (CTF) evaluation used · 1 source · 1 quote
Dedicated German evaluation suite used · 1 source · 1 quote
Disabling tool search in evaluation used · 1 source · 1 quote
Domain whitelist for agent environments used · 1 source · 1 quote
Domain-specific evaluation suites used · 1 source · 1 quote
EvalPlus-sanitized benchmark variants used · 1 source · 1 quote
Execution-grounded evaluation used · 1 source · 1 quote
GPU kernel optimization task suite used · 1 source · 1 quote
Held-out validation set construction used · 1 source · 1 quote
KernelBench used · 1 source · 1 quote
LongBench v2 used · 1 source · 1 quote
PaperBench hierarchical rubric evaluation used · 1 source · 1 quote
PinchBench used · 1 source · 1 quote
ProfBench with Search used · 1 source · 1 quote
SQuAD used · 1 source · 1 quote
SWE-bench Verified evaluation used · 1 source · 1 quote
Three-configuration cyber range testing used · 1 source · 1 quote
Three-shot prompting for MBPP-Sanitized used · 1 source · 1 quote
Token-F1 grading for retrieval answers used · 1 source · 1 quote
Vals.ai used · 1 source · 1 quote
Verified CUDA kernels for synthetic data used · 1 source · 1 quote
Zero-shot prompting for HumanEval used · 1 source · 1 quote
CAD visual reproduction optional · 1 source · 1 quote
Computer-use closed-loop evaluation optional · 1 source · 1 quote
Artificial Analysis Intelligence Index evaluated · 1 source · 1 quote
RoboLab evaluated · 1 source · 1 quote
RoboArena mentioned · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelmethods
GLM-5.3-Flash optionalCAD visual reproduction optionalComputer-use closed-loop evaluation optional—
GLM-5.3 usedDisabling tool search in evaluation usedDomain whitelist for agent environments used—
NVIDIA-Nemotron-3-Ultra-550B-A55B usedBrowseComp with Search usedKernelBench usedLongBench v2 usedPinchBench usedProfBench with Search usedScaleAI Multi Challenge Multi Turn Instruction Following usedSQuAD usedVals.ai used—
DeepSeek-V3.2 usedF2P/P2F acceptance criterion for executable environments used—
FLUX 3 Action evaluatedRoboLab evaluatedRoboArena mentioned—
Kimi K3 coreLiving in-house benchmark suite coreGPU kernel optimization task suite used—
Kolibri-1 defaultAutomatic checkpoint evaluation across capability benchmarks defaultDedicated German evaluation suite usedDomain-specific evaluation suites usedToken-F1 grading for retrieval answers used—
Laguna-S-2.1 usedExecution-grounded evaluation used—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B usedEvalPlus-sanitized benchmark variants usedFive-shot chain-of-thought prompting for GPQA Diamond usedThree-shot prompting for MBPP-Sanitized usedVerified CUDA kernels for synthetic data usedZero-shot prompting for HumanEval usedArtificial Analysis Intelligence Index evaluated—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B usedEvalPlus-sanitized benchmark variants usedFive-shot chain-of-thought prompting for GPQA Diamond usedThree-shot prompting for MBPP-Sanitized usedZero-shot prompting for HumanEval used—
X-VLA usedHeld-out validation set construction used—
gpt-oss-120b usedCapture-the-Flag (CTF) evaluation usedPaperBench hierarchical rubric evaluation usedSWE-bench Verified evaluation usedThree-configuration cyber range testing used—
↑