Open Models Almanac
Methodsevaluation

methods · outside the pipeline

evaluation

195 methods filed at this node or below it, from the sources of 34 models.

In this branch 195

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 69

Prompt-based output standardization used · 5 sources · 5 quotes
Evaluation without safety filters used · 3 sources · 3 quotes
Refusal-suppressed capability evaluation used · 3 sources · 3 quotes
Chain-of-thought few-shot prompting used · 2 sources · 3 quotes
Controlled evaluation with fixed settings used · 2 sources · 2 quotes
Open-loop evaluation used · 2 sources · 2 quotes
63 single mentions
Benchmark decontamination core · 1 source · 1 quote
Five-shot in-context evaluation default · 1 source · 1 quote
avg@3 used · 1 source · 1 quote
avg@k used · 1 source · 1 quote
Best-performing checkpoint selection used · 1 source · 1 quote
Candidate-pair overlap gate used · 1 source · 1 quote
Capability-partition stratification used · 1 source · 1 quote
Controlled proxy-scale ablation used · 1 source · 1 quote
Coverage scoring by union across revisions used · 1 source · 1 quote
DAG critical-path latency measurement used · 1 source · 1 quote
Distributional correctness score used · 1 source · 1 quote
Evaluation trajectory release used · 1 source · 1 quote
Evaluation-setting documentation used · 1 source · 1 quote
Explicit abstention option used · 1 source · 1 quote
Exploit development evaluation used · 1 source · 1 quote
Fixed MTP acceptance length used · 1 source · 1 quote
Flash-SFT proxy evaluation used · 1 source · 1 quote
Four-run mean pass@1 used · 1 source · 1 quote
Grounding and hallucination evaluation used · 1 source · 1 quote
Hard same-domain negative examples used · 1 source · 1 quote
IDF-weighted n-gram matching used · 1 source · 2 quotes
Joint validation protocol used · 1 source · 1 quote
Loose-match accuracy used · 1 source · 1 quote
Official-source external baselines used · 1 source · 1 quote
Overlapping byte-window chunking used · 1 source · 1 quote
Pre-marked object-location evaluation used · 1 source · 1 quote
Prompt-enforced answering used · 1 source · 1 quote
ProtocolQA robustness validation used · 1 source · 1 quote
Question-weighted IDF overlap thresholding used · 1 source · 1 quote
Randomly sampled five-shot evaluation used · 1 source · 1 quote
Refusal behavior quantification used · 1 source · 1 quote
Rollout non-termination tracking used · 1 source · 1 quote
Rollout-based audit used · 1 source · 1 quote
Score equivalence threshold used · 1 source · 1 quote
Sentence masking importance estimation used · 1 source · 1 quote
Separate augmentation and judge models used · 1 source · 1 quote
Standard function-calling format used · 1 source · 1 quote
Standardized agent evaluation protocol used · 1 source · 1 quote
Standardized prompts and task settings used · 1 source · 1 quote
Structured True/False LLM evaluation used · 1 source · 1 quote
Task-specific progress scoring rubric used · 1 source · 1 quote
Test-based injected-bug filtering used · 1 source · 1 quote
Two-dimensional coding-agent evaluation used · 1 source · 1 quote
Visual checking of generated renders used · 1 source · 1 quote
Vulnerability discovery evaluation used · 1 source · 1 quote
Zero-shot baseline fallback used · 1 source · 1 quote
Additive Gaussian and shot-noise perturbations evaluated · 1 source · 1 quote
Controlled agent-scaffold comparison evaluated · 1 source · 1 quote
Pretraining and fine-tuning ablation comparison evaluated · 1 source · 1 quote
V-model testing and validation mentioned · 1 source · 1 quote

cybersecurity - vulnerability discovery; exploit development 4

all mentioned once
Cybersecurity guardrails core · 1 source · 1 quote
Exploit and functional-correctness testing used · 1 source · 1 quote
Transient build and package cache purging used · 1 source · 1 quote

benchmark 33

all mentioned once
Living in-house benchmark suite core · 1 source · 1 quote
BrowseComp with Search used · 1 source · 1 quote
Capture-the-Flag (CTF) evaluation used · 1 source · 1 quote
Dedicated German evaluation suite used · 1 source · 1 quote
Disabling tool search in evaluation used · 1 source · 1 quote
Domain whitelist for agent environments used · 1 source · 1 quote
Domain-specific evaluation suites used · 1 source · 1 quote
EvalPlus-sanitized benchmark variants used · 1 source · 1 quote
Execution-grounded evaluation used · 1 source · 1 quote
GPU kernel optimization task suite used · 1 source · 1 quote
Held-out validation set construction used · 1 source · 1 quote
KernelBench used · 1 source · 1 quote
LongBench v2 used · 1 source · 1 quote
PaperBench hierarchical rubric evaluation used · 1 source · 1 quote
PinchBench used · 1 source · 1 quote
ProfBench with Search used · 1 source · 1 quote
SQuAD used · 1 source · 1 quote
SWE-bench Verified evaluation used · 1 source · 1 quote
Three-configuration cyber range testing used · 1 source · 1 quote
Three-shot prompting for MBPP-Sanitized used · 1 source · 1 quote
Token-F1 grading for retrieval answers used · 1 source · 1 quote
Vals.ai used · 1 source · 1 quote
Verified CUDA kernels for synthetic data used · 1 source · 1 quote
Zero-shot prompting for HumanEval used · 1 source · 1 quote
CAD visual reproduction optional · 1 source · 1 quote
Computer-use closed-loop evaluation optional · 1 source · 1 quote
Artificial Analysis Intelligence Index evaluated · 1 source · 1 quote
RoboLab evaluated · 1 source · 1 quote
RoboArena mentioned · 1 source · 1 quote

evaluation harness 21

NeMo Evaluator SDK used · 3 sources · 3 quotes
DeepSeek Harness Minimal mode used · 2 sources · 2 quotes
Harbor Framework used · 2 sources · 2 quotes
18 single mentions
Savanna Benchmark abstraction core · 1 source · 1 quote
Atomic binary rubric evaluation used · 1 source · 1 quote
Benchmark-specific evaluation recipes used · 1 source · 1 quote
Custom internal evaluation harness used · 1 source · 1 quote
Isolated-container-per-rollout execution used · 1 source · 1 quote
JSON answer-format prompting used · 1 source · 1 quote
LM Evaluation Harness used · 1 source · 1 quote
mini-swe-agent harness used · 1 source · 1 quote
NeMo Gym and NeMo Evaluator harness used · 1 source · 1 quote
NeMo Skills used · 1 source · 1 quote
Pool agent harness used · 1 source · 1 quote
Priority-ordered answer extraction parser used · 1 source · 1 quote
Tinker Cookbook benchmark framework used · 1 source · 1 quote
BenchmarkEvaluator optional · 1 source · 1 quote

judge 30

LLM-as-a-Judge used · 2 sources · 2 quotes
29 single mentions
Agent-as-a-Judge used · 1 source · 2 quotes
Agent-as-a-Verifier used · 1 source · 1 quote
Agentic judge protocol used · 1 source · 1 quote
Claude Opus 4.6 judge used · 1 source · 1 quote
Deterministic tool-call verifier used · 1 source · 1 quote
Frozen judge model used · 1 source · 1 quote
GPT-5.5 (medium) judge used · 1 source · 1 quote
Human-calibrated LLM-as-a-judge used · 1 source · 1 quote
JSON-escaped input formatting for judge used · 1 source · 1 quote
Judge-LLM ensemble scoring used · 1 source · 2 quotes
Large reasoning model as evaluator used · 1 source · 1 quote
LLM autograding with expert validation used · 1 source · 1 quote
LLM judge for conversation classification used · 1 source · 1 quote
LLM judge for document-quality annotation used · 1 source · 1 quote
LLM-as-a-Verifier used · 1 source · 1 quote
LLM-based correctness judging used · 1 source · 1 quote
LLM-based external API usage inspection used · 1 source · 1 quote
Official task verifier scoring used · 1 source · 1 quote
On-policy tie-breaking used · 1 source · 1 quote
Order-swapped verifier evaluation used · 1 source · 1 quote
Probabilistic Pivot Tournament used · 1 source · 1 quote
Reward-hack judge used · 1 source · 1 quote
Rule-based anti-cheat checks used · 1 source · 1 quote
Three-criterion trajectory verification used · 1 source · 1 quote

averaging - mean@k; avg@k; pass@k; pass^k 5

all mentioned once
4-shot sampled pass@1 used · 1 source · 1 quote
Mean benchmark score used · 1 source · 1 quote
Mean of per-rubric scores used · 1 source · 1 quote
Pass@k used · 1 source · 1 quote
Sampled pass@1 used · 1 source · 1 quote

human & real-world evaluation 33

Blind expert evaluation of engineering tasks used · 3 sources · 3 quotes
Multi-turn open-ended external red-teaming used · 3 sources · 3 quotes
31 single mentions
Held-out evaluation benchmarks default · 1 source · 1 quote
Automatic success-rate measurement used · 1 source · 1 quote
Blind expert judging used · 1 source · 1 quote
Browser-based testing and iterative repair used · 1 source · 1 quote
Closed-loop simulation used · 1 source · 1 quote
Customer-specific contextualized evaluation used · 1 source · 2 quotes
Cyber range exercises used · 1 source · 1 quote
Dangerous-capability uplift assessment used · 1 source · 1 quote
Human evaluation used · 1 source · 1 quote
Human-verified and LLM-based evaluation used · 1 source · 1 quote
Interleaved policy evaluation used · 1 source · 1 quote
Internal engineering-task evaluation used · 1 source · 2 quotes
On-vehicle road testing used · 1 source · 1 quote
Pairwise comparison for Chinese writing used · 1 source · 1 quote
Pre-release safety evaluation used · 1 source · 1 quote
Real-world feedback refinement used · 1 source · 1 quote
Repeated trials for statistical robustness used · 1 source · 1 quote
Same-prompt comparative DevOps testing used · 1 source · 1 quote
Turn-level blind A/B testing used · 1 source · 1 quote
White-collar enterprise-task evaluation used · 1 source · 1 quote
Blind real-robot testing evaluated · 1 source · 1 quote
A/B testing on real-world tasks mentioned · 1 source · 1 quote
Continuous monitoring mentioned · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelfiled herecybersecurity - vulnerability discovery; exploit developmentbenchmarkevaluation harnessjudgeaveraging - mean@k; avg@k; pass@k; pass^khuman & real-world evaluation
GLM-5.3-Flash optional——CAD visual reproduction optionalComputer-use closed-loop evaluation optional—————
DeepSeek-V4.1-Flash usedDAG critical-path latency measurement usedScore equivalence threshold usedStandardized agent evaluation protocol usedControlled agent-scaffold comparison evaluated—Internet restriction and Git-history removal usedTransient build and package cache purging used——DeepSeek Harness Minimal mode used————
Hy4-preview used——————Blind expert evaluation of engineering tasks used—
GLM-5.3 usedavg@3 usedCoverage scoring by union across revisions usedTwo-dimensional coding-agent evaluation usedZero-shot baseline fallback used——Disabling tool search in evaluation usedDomain whitelist for agent environments used—Isolated-container-per-rollout execution usedmini-swe-agent harness used—LLM-as-a-Judge usedLLM-based external API usage inspection usedOfficial task verifier scoring usedRule-based anti-cheat checks used——Real-world codebase testing with security teams used—
DeepSeek-V4-Flash-0731 mentioned——————A/B testing on real-world tasks mentioned—
Hy3 used——————Real-world feedback refinement used—
MiMo-V2.6-Flash usedBehavioral testing with stubs and edge cases usedRollout-based audit used———Atomic binary rubric evaluation used———Same-prompt comparative DevOps testing used—
NVIDIA-Nemotron-3-Ultra-550B-A55B default——BrowseComp with Search usedKernelBench usedLongBench v2 usedPinchBench usedProfBench with Search usedScaleAI Multi Challenge Multi Turn Instruction Following usedSQuAD usedVals.ai used—Harbor Framework usedNeMo Evaluator SDK usedNeMo Skills used—Deterministic tool-call verifier usedLLM-based correctness judging used——Held-out evaluation benchmarks defaultAutomated red-teaming for indirect prompt injection used—
Alpamayo 1.5 / 2 Super usedStructured True/False LLM evaluation used————Large reasoning model as evaluator used——Closed-loop simulation usedHuman-verified and LLM-based evaluation usedOn-vehicle road testing used—
DeepSeek-V3.2 usedSample-test and length-constraint filtering usedStandard function-calling format used——F2P/P2F acceptance criterion for executable environments used—————
DeepSeek-V4-Flash-Vision-Exp used———DeepSeek Harness Minimal mode used————
DeepSeek-V4-Pro used——————Internal engineering-task evaluation usedPairwise comparison for Chinese writing usedWhite-collar enterprise-task evaluation used—
FLUX 3 Action evaluated——RoboLab evaluatedRoboArena mentioned————Blind real-robot testing evaluated—
GLM-5.2 used————Agent-as-a-Judge usedGPT-5.5 (medium) judge used———
GR00T N1.7 usedSimulation and real-robot benchmark evaluation used————Multimodal LLM instruction-adherence judging used——Automatic success-rate measurement usedHuman-observer evaluation of real-robot tasks used—
Gemma 4 31B usedEvaluation without safety filters used——————Human evaluation usedContinuous monitoring mentioned—
Inkling usedEvaluation-setting documentation usedRefusal-suppressed capability evaluation used———Tinker Cookbook benchmark framework usedBenchmarkEvaluator optional———Dangerous-capability uplift assessment usedMulti-turn open-ended external red-teaming usedPre-release safety evaluation used—
Kimi K3 coreExploit development evaluation usedPrompt-enforced answering usedSandboxed GPU-kernel optimization evaluation usedTrajectory analysis for failure attribution usedTwo-tier cybersecurity capability evaluation usedVulnerability discovery evaluation used——Living in-house benchmark suite coreGPU kernel optimization task suite used——Agentic judge protocol used——Blind expert judging used—
Kolibri-1 coreControlled proxy-scale ablation usedDistributional correctness score usedExplicit abstention option usedFlash-SFT proxy evaluation usedGrounding and hallucination evaluation usedIDF-weighted n-gram matching usedQuestion-weighted IDF overlap thresholding usedRecall-calibrated decontamination threshold usedSentence masking importance estimation usedSeparate question and answer n-gram indexes usedStandardized prompts and task settings usedTest-based injected-bug filtering used——Automatic checkpoint evaluation across capability benchmarks defaultDedicated German evaluation suite usedDomain-specific evaluation suites usedToken-F1 grading for retrieval answers used—Savanna Benchmark abstraction coreWeb-fetch removal from evaluation harnesses used—Frozen judge model usedLLM judge for conversation classification usedLLM judge for document-quality annotation usedSchema-constrained decoding for judge outputs used—Mean benchmark score usedPass@k used—Customer-proxy evaluation without customer data training coreCustomer-specific contextualized evaluation usedInternal checkpoint preview and employee feedback usedTurn-level blind A/B testing used—
LFM3-1.7B defaultFive-shot in-context evaluation defaultChain-of-thought few-shot prompting usedLoose-match accuracy usedRandomly sampled five-shot evaluation usedRepeated evaluation with answer-order permutations used———Custom internal evaluation harness usedPriority-ordered answer extraction parser usedRobust answer parsing in a custom evaluation harness used—Judge-LLM ensemble scoring usedOn-policy tie-breaking used———
Laguna-S-2.1 usedavg@k usedDiverse benchmark validation for quantization usedEvaluation trajectory release usedFour-run mean pass@1 usedOfficial-source external baselines used——Execution-grounded evaluation used—Automatic in-flight checkpoint evaluation scheduling usedHarbor Framework usedPool agent harness used—Decompositional instruction-following judge usedHuman-calibrated LLM-as-a-judge usedReward-hack judge used———
LingBot-VLA 2.0 usedOpen-loop evaluation optionalAdditive Gaussian and shot-noise perturbations evaluated——————Randomized-order sequential model evaluation usedRepeated trials for statistical robustness used—
MiMo-V2.6-Pro usedBehavioral testing with stubs and edge cases usedRollout-based audit usedVisual checking of generated renders used———Atomic binary rubric evaluation used———Same-prompt comparative DevOps testing used—
MiniMax-M3 used————Agent-as-a-Verifier usedLLM-as-a-Judge used———
Mistral-Large-4 core—Cybersecurity guardrails coreExploit and functional-correctness testing used——Vulnerability test harness validation against unpatched and fixed code used———Browser-based testing and iterative repair used—
MolmoAct 2 usedBest-performing checkpoint selection usedPre-marked object-location evaluation used———————
NVIDIA-Nemotron-3.5-Lightning-30B-A3B coreBenchmark decontamination coreCandidate-pair overlap gate usedCapability-partition stratification usedChain-of-thought few-shot prompting usedContext-length and candidate-count stratification usedControlled evaluation with fixed settings usedHard same-domain negative examples usedOverlapping byte-window chunking usedRollout non-termination tracking usedSeparate augmentation and judge models usedSynthetic benchmark-question perturbation evaluation usedMatched supervised fine-tuning checkpoint comparison evaluatedV-model testing and validation mentioned——EvalPlus-sanitized benchmark variants usedFive-shot chain-of-thought prompting for GPQA Diamond usedThree-shot prompting for MBPP-Sanitized usedVerified CUDA kernels for synthetic data usedZero-shot prompting for HumanEval usedArtificial Analysis Intelligence Index evaluated—LM Evaluation Harness usedNeMo Evaluator SDK usedNeMo Gym and NeMo Evaluator harness used—Claude Opus 4.6 judge usedJSON-escaped input formatting for judge usedLLM-as-a-Verifier usedLLM-judge verification of contamination pairs usedOrder-swapped verifier evaluation usedProbabilistic Pivot Tournament usedTeacher-based text-only equivalence judging usedThree-criterion trajectory verification used—4-shot sampled pass@1 usedMean of per-rubric scores usedSampled pass@1 used——
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B coreBenchmark decontamination coreCandidate-pair overlap gate usedCapability-partition stratification usedChain-of-thought few-shot prompting usedContext-length and candidate-count stratification usedControlled evaluation with fixed settings usedHard same-domain negative examples usedOverlapping byte-window chunking usedRollout non-termination tracking usedSeparate augmentation and judge models usedSynthetic benchmark-question perturbation evaluation usedMatched supervised fine-tuning checkpoint comparison evaluated——EvalPlus-sanitized benchmark variants usedFive-shot chain-of-thought prompting for GPQA Diamond usedThree-shot prompting for MBPP-Sanitized usedZero-shot prompting for HumanEval used—Benchmark-specific evaluation recipes usedLM Evaluation Harness usedNeMo Evaluator SDK used—Claude Opus 4.6 judge usedJSON-escaped input formatting for judge usedLLM-as-a-Verifier usedLLM-judge verification of contamination pairs usedOrder-swapped verifier evaluation usedProbabilistic Pivot Tournament usedTeacher-based text-only equivalence judging usedThree-criterion trajectory verification used—4-shot sampled pass@1 usedMean of per-rubric scores usedSampled pass@1 used——
Qwen3.5-397B-A17B usedPrompt-based output standardization used———JSON answer-format prompting used————
Qwen3.6-35B-A3B usedPrompt-based output standardization used———————
Qwen3.8-Flash-Next usedFixed MTP acceptance length usedJoint validation protocol used———————
X-VLA usedOpen-loop evaluation used——Held-out validation set construction used—————
gpt-oss-120b usedProtocolQA robustness validation usedRefusal behavior quantification used——Capture-the-Flag (CTF) evaluation usedPaperBench hierarchical rubric evaluation usedSWE-bench Verified evaluation usedThree-configuration cyber range testing used——LLM autograding with expert validation used——Cyber range exercises usedExternal expert review of safety evaluation methodology used—
π0.5 usedTask-specific progress scoring rubric usedPretraining and fine-tuning ablation comparison evaluated——————Interleaved policy evaluation usedZero-shot evaluation in unseen environments used—

Proposed children

Paths the corpus wanted and the taxonomy does not have. A human promotes them into the outline; the extraction cannot.

benchmark contamination detectionbenchmark decontamination, candidate retrieval, lexical overlap, semantic overla
↑