evaluation

Model	filed here	benchmark	evaluation harness	judge	human & real-world evaluation
GLM-5.3-Flash optional	—	CAD visual reproduction optionalComputer use closed-loop evaluation optional—	—	—	—
DeepSeek-V4.1-Flash used	Almost@1 usedDeadline-bounded rollout evaluation usedEvaluation equivalence threshold usedMean@5 usedStandardized agent evaluation protocol usedModel–harness co-design optionalSingle-agent and multi-agent evaluation configurations evaluated—	Fail-to-pass and pass-to-pass evaluation points usedHigh-confidence ProgramBench subset usedMultiple-rollout evaluation used—	Cross-scaffold evaluation usedDeepSeek Harness Minimal mode used—	Independent quality-inspection agent used—	Reward hacking mitigation in evaluations usedSystematic robustness-boundary characterization used—
Hy4-preview used	—	—	—	—	Blind side-by-side expert evaluation used—
DeepSeek-V4-Flash-0731 mentioned	—	—	—	—	A/B testing on real tasks mentioned—
NVIDIA-Nemotron-3-Ultra-550B-A55B default	—	BrowseComp with Search usedKernelBench usedLongBench v2 usedPinchBench usedProfBench with Search usedScaleAI Multi Challenge Multi Turn Instruction Following usedSQuAD usedVals.ai used—	Harbor usedNeMo Evaluator SDK usedNeMo Skills used—	Deterministic tool-call verifier usedLLM-based correctness judging used—	Held-out evaluation defaultAutomated red-teaming loop used—
MiMo-V2.6-Flash used	Behavior testing with stubs and edge cases usedRollout-based auditing used—	—	Atomic binary rubric evaluation used—	—	Same-prompt comparative DevOps testing used—
GLM-5.3 used	avg@k usedTwo-dimensional coding-agent evaluation usedUnion-of-capabilities coverage scoring usedZero-shot base-model baseline fallback used—	Disabling tool search usedDomain whitelist used—	Isolated container per rollout usedmini-swe-agent harness used—	LLM-based anti-cheating judgment usedLLM-based external API usage inspection usedOfficial task verifier scoring usedRule-based anti-cheat checks used—	Real-world codebase testing used—
Hy3 used	—	—	—	—	Real-world feedback refinement used—
GLM-5.2 used	—	—	—	Agent-as-a-Judge usedGPT-5.5 (medium) judge model used—	—
MiniMax-M3 used	—	—	—	Agent-as-a-Verifier usedLLM-as-a-Judge used—	—
DeepSeek-V3.2 used	Sample test and length-constraint filtering usedStandard function-calling format used—	F2P/P2F acceptance criterion used—	—	—	—
DeepSeek-V4-Flash-Vision-Exp used	—	—	DeepSeek Harness Minimal mode used—	—	—
DeepSeek-V4-Pro used	—	—	—	—	Internal engineering task evaluation usedPairwise comparison for Chinese writing usedWhite-collar enterprise task evaluation used—
Gemma 4 31B used	Evaluation without safety filters used—	—	—	—	Human evaluation usedContinuous monitoring mentioned—
Inkling used	Documenting evaluation configuration usedRefusal-suppressed variants for capability estimation used—	—	Benchmark evaluation framework usedInline training evaluators optional—	—	Dangerous-capability uplift assessment usedMulti-turn open-ended external red-teaming usedPre-release safety evaluation used—
Kimi K3 core	End-to-end exploit development evaluation usedPrompt engineering to elicit answers usedSandboxed GPU kernel optimization evaluation usedTrajectory analysis usedTwo-tier cybersecurity capability evaluation usedVulnerability discovery with proof-of-concept evaluation used—	Living in-house benchmark suite coreGPU kernel optimization task suite used—	—	Mandatory agentic judge protocol used—	Blind side-by-side expert evaluation used—
Laguna-S-2.1 used	avg@k usedDiverse benchmark validation for quantization usedFour-run mean pass@1 usedFull trajectory release usedOfficial-release-source baseline restriction used—	Execution-grounded evaluation used—	Automatic in-flight checkpoint evaluation scheduling usedHarbor usedPool used—	Decompositional instruction-following judge usedHuman-calibrated LLM-as-a-Judge usedReward-hack detection judge used—	—
MiMo-V2.6-Pro used	Behavior testing with stubs and edge cases usedRollout-based auditing usedVisual checking of generated renders used—	—	Atomic binary rubric evaluation used—	—	Same-prompt comparative DevOps testing used—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B used	V-model iterative testing and validation mentioned—	Verified CUDA kernels usedArtificial Analysis Intelligence Index evaluated—	NeMo Gym and NeMo Evaluator-based harness used—	—	—
Qwen3.5-397B-A17B used	Prompt-based output standardization used—	—	JSON-structured answer-format prompt used—	—	—
Qwen3.6-35B-A3B used	Prompt-based output standardization used—	—	—	—	—
Qwen3.8-Flash-Next used	Fixed MTP draft-token acceptance length usedJoint validation protocol used—	—	—	—	—
gpt-oss-120b used	ProtocolQA robustness validation usedRefusal behavior quantification used—	Capture-the-Flag (CTF) evaluation usedPaperBench usedSWE-bench Verified usedThree-configuration cyber range testing used—	—	LLM autograding with expert validation used—	Cyber range exercises usedExternal expert review of safety evaluation methodology used—

In this branch 109

filed here 35

benchmark 25

evaluation harness 14

judge 16

human & real-world evaluation 19

By model