Open Models Almanac
Methodsinference & serving

methods · pipeline stage 6 of 6

inference & serving

521 methods filed at this node or below it, from the sources of 45 models.

In this branch 521

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 9

Pre-deployment robot-action safety checks used · 2 sources · 2 quotes
7 single mentions
Chunk fusion core · 1 source · 1 quote
Communication optimization used · 1 source · 1 quote
Remote VLA inference server used · 1 source · 1 quote
Interleaved text-and-audio generation optional · 1 source · 1 quote
Offline generation optional · 1 source · 1 quote
Reusable-cache optimization optional · 1 source · 1 quote
Server-client policy inference optional · 1 source · 1 quote

decoding strategy 52

Multi-Token Prediction core · 19 sources · 19 quotes
Speculative decoding core · 15 sources · 15 quotes
DSpark core · 14 sources · 14 quotes
DFlash core · 6 sources · 7 quotes
Five-layer SWA MTP drafter core · 2 sources · 2 quotes
Three-token speculative draft depth core · 2 sources · 2 quotes
Greedy decoding default · 4 sources · 4 quotes
Standardized sampling configuration default · 3 sources · 3 quotes
EAGLE used · 10 sources · 10 quotes
Multi-layer EAGLE used · 5 sources · 5 quotes
NEXTN speculative decoding used · 4 sources · 4 quotes
Best-of-N selection used · 3 sources · 4 quotes
Action-chunk execution used · 2 sources · 2 quotes
Multi-stage candidate filtering used · 2 sources · 2 quotes
Presence penalty tuning used · 2 sources · 3 quotes
Standalone parallel speculative drafters used · 2 sources · 3 quotes
Speculative sampling optional · 3 sources · 3 quotes
Task-specific sampling-parameter presets optional · 3 sources · 3 quotes
LiLiCorr optional · 2 sources · 2 quotes
32 from one source, or only mentioned as not used
Confidence-scheduled verification core · 1 source · 1 quote
Fused recurrent replay kernel core · 1 source · 1 quote
Recurrent Markov draft head core · 1 source · 1 quote
Recursive shared MTP-head drafting core · 1 source · 2 quotes
Semi-autoregressive draft generation core · 1 source · 1 quote
Three-layer dense MLA speculative drafter core · 1 source · 1 quote
Top-K candidate lattice core · 1 source · 1 quote
Same-checkpoint target and draft weights default · 1 source · 1 quote
Block verification used · 1 source · 1 quote
Clustered top-K projection used · 1 source · 1 quote
Drafter-target KV-cache sharing used · 1 source · 1 quote
Grammar-constrained decoding used · 1 source · 1 quote
LiLiCorr single-pass correlation head used · 1 source · 1 quote
Longest-thinking-trace selection used · 1 source · 1 quote
MTP-1 speculative decoding used · 1 source · 1 quote
MTP-to-EAGLE-3 draft-model fine-tuning used · 1 source · 1 quote
Recurrent conditioning head used · 1 source · 1 quote
Second-most-likely action querying used · 1 source · 1 quote
Throughput-based draft-block sizing used · 1 source · 1 quote
Top-p and top-k sampling used · 1 source · 1 quote
Unique mask tokens used · 1 source · 1 quote
Autoregressive discrete action-token inference optional · 1 source · 1 quote
Chat Prefix Completion optional · 1 source · 1 quote
DSpark draft head optional · 1 source · 1 quote
MTP draft-length configuration optional · 1 source · 1 quote
Probabilistic draft sampling optional · 1 source · 1 quote
Temperature sampling for bimanual tasks optional · 1 source · 1 quote
Concurrency-aware draft-length tuning evaluated · 1 source · 1 quote
Confidence head not used · 1 source · 1 quote
Dynamic action querying not used · 1 source · 1 quote

reasoning control 49

Thinking mode core · 49 sources · 55 quotes
Continuously controllable reasoning effort core · 3 sources · 6 quotes
Interleaved thinking core · 3 sources · 3 quotes
Adaptive reasoning core · 2 sources · 2 quotes
Always-on thinking mode core · 2 sources · 2 quotes
Full chain-of-thought output core · 2 sources · 2 quotes
Chain-of-thought reasoning default · 6 sources · 6 quotes
Reasoning enabled by default default · 4 sources · 4 quotes
clear_thinking chat-template flag default · 2 sources · 2 quotes
Max thinking mode with self-set compute budget default · 2 sources · 3 quotes
System-prompt reasoning-effort conditioning default · 2 sources · 2 quotes
Reasoning token-budget control used · 7 sources · 8 quotes
Persistent reasoning history used · 3 sources · 3 quotes
Generate-verify-refine used · 2 sources · 2 quotes
Reasoning parser used · 2 sources · 2 quotes
Disable reasoning via enable_thinking optional · 5 sources · 5 quotes
Parallel test-time compute scaling evaluated · 2 sources · 2 quotes
Parallel-fewest-step sampling evaluated · 2 sources · 2 quotes
31 from one source, or only mentioned as not used
control-token-enabled thinking mode core · 1 source · 1 quote
Hierarchical subtask inference core · 1 source · 2 quotes
Chat-template thinking-mode switch default · 1 source · 1 quote
Preset effort tiers mapped to scalar values default · 1 source · 2 quotes
Reasoning cannot be disabled default · 1 source · 1 quote
Generation-prefix mode selection used · 1 source · 1 quote
Length-dependent completion-budget signal used · 1 source · 1 quote
Stage-wise effort-budget curriculum used · 1 source · 1 quote
Step3p5 reasoning parser used · 1 source · 1 quote
Test-time compute scaling used · 1 source · 1 quote
Think-tag response formatting used · 1 source · 1 quote
Turn budget capping used · 1 source · 1 quote
Turn-aware prompting used · 1 source · 1 quote
Per-request thinking control optional · 1 source · 1 quote
Reasoning-enabled boolean toggle optional · 1 source · 1 quote
Step-by-step thinking mode optional · 1 source · 1 quote
Task- and mode-specific sampling parameters optional · 1 source · 2 quotes
Task-appropriate maximum output length optional · 1 source · 1 quote
effort parameters mentioned · 1 source · 1 quote
Qwen3 soft thinking switch not used · 2 sources · 2 quotes
Reasoning effort configuration availability not used · 1 source · 1 quote

KV cache management 53

Prefix caching core · 10 sources · 10 quotes
Compressed KV caching core · 3 sources · 3 quotes
Cross-layer KV-cache sharing core · 2 sources · 2 quotes
Hybrid-state caching core · 2 sources · 2 quotes
SWA Bounded Replay core · 2 sources · 2 quotes
Chunked prefill default · 6 sources · 6 quotes
Prompt caching used · 4 sources · 4 quotes
Language-model-only serving mode used · 3 sources · 3 quotes
Internal prefill checkpointing used · 2 sources · 2 quotes
RadixCache used · 2 sources · 2 quotes
Periodic cache checkpointing optional · 3 sources · 3 quotes
HiCache mentioned · 2 sources · 2 quotes
40 single mentions
Block-based KV cache core · 1 source · 1 quote
Customized heterogeneous KV-cache layout core · 1 source · 1 quote
KDA-aware prefix-cache management core · 1 source · 2 quotes
Persistent per-dialogue-context KV caching core · 1 source · 1 quote
Prefix reuse core · 1 source · 1 quote
Shared-free-list cache allocation core · 1 source · 1 quote
Write-back external KV-cache policy core · 1 source · 1 quote
Automatic cache support default · 1 source · 1 quote
Asynchronous KV-cache offloading used · 1 source · 1 quote
Disabling prefix caching for benchmarking used · 1 source · 1 quote
KDA with prefill cache used · 1 source · 1 quote
KV cache sharing used · 1 source · 1 quote
KV-cache reset on weight synchronization used · 1 source · 1 quote
Overlapped host-memory prefetching used · 1 source · 1 quote
Pre-scheduled tile chunking used · 1 source · 1 quote
Preallocated static KV cache used · 1 source · 1 quote
Request-level prefix cache used · 1 source · 1 quote
Stateless in-memory prefix caching used · 1 source · 1 quote
You Only Cache Once (YOCO) used · 1 source · 1 quote
Decode context parallelism optional · 1 source · 1 quote
KV-cache management optional · 1 source · 1 quote
Zero SWA caching optional · 1 source · 1 quote
8-bit Mamba cache quantization evaluated · 1 source · 1 quote
KV-cache offloading evaluated · 1 source · 1 quote
Mamba prefix caching in align mode evaluated · 1 source · 1 quote

inference quantization 112

Quantization core · 4 sources · 4 quotes
FP4 global KV-cache quantization core · 2 sources · 2 quotes
Mobile quantization core · 2 sources · 2 quotes
Per-tensor FP8 quantization core · 2 sources · 2 quotes
FP8 default · 18 sources · 18 quotes
BF16 inference default · 2 sources · 2 quotes
Mixed-precision NVFP4 routed-expert quantization default · 2 sources · 2 quotes
FP8 KV-cache quantization used · 12 sources · 13 quotes
Post-training quantization used · 5 sources · 5 quotes
NVFP4 quantization used · 4 sources · 4 quotes
MSE-based weight-scale selection used · 3 sources · 4 quotes
MXFP4 weight quantization used · 3 sources · 3 quotes
Four-Over-Six used · 2 sources · 3 quotes
GGUF used · 2 sources · 2 quotes
Mamba SSM-cache quantization used · 2 sources · 2 quotes
ModelOpt NVFP4 re-quantization used · 2 sources · 2 quotes
MXFP8 activation quantization used · 2 sources · 2 quotes
FP8 E4M3 quantization evaluated · 2 sources · 2 quotes
Max-based weight scaling evaluated · 2 sources · 2 quotes
92 from one source, or only mentioned as not used
Backbone-coupled MTP KV-cache precision core · 1 source · 1 quote
Dynamic activation scaling core · 1 source · 1 quote
Mixed-FP8 layers in an NVFP4 recipe core · 1 source · 1 quote
Mixed-precision inference core · 1 source · 1 quote
Native 2-bit quantization core · 1 source · 1 quote
NVFP4 routed-expert GEMM quantization core · 1 source · 1 quote
Offline weight-layout permutation core · 1 source · 1 quote
FP8 latent KV-cache quantization default · 1 source · 2 quotes
NVFP4 KV-cache quantization default · 1 source · 2 quotes
Abs-max quantization used · 1 source · 1 quote
AWQ INT4 weight-only quantization used · 1 source · 1 quote
Channel-wise quantization used · 1 source · 1 quote
Component-wise precision selection used · 1 source · 1 quote
Embedding and KV-cache quantization used · 1 source · 1 quote
Flex_AWQ_SSZ used · 1 source · 1 quote
FP16 multimodal projector used · 1 source · 1 quote
FP4 attention-indexer precision used · 1 source · 1 quote
FP8 precision used · 1 source · 1 quote
FP8-or-higher lm_head precision used · 1 source · 1 quote
Layer-wise mixed INT4/INT8 quantization used · 1 source · 1 quote
ModelOpt FP4 quantization used · 1 source · 1 quote
MoE-layer quantization used · 1 source · 1 quote
MXFP4 block-scaled tensor packing used · 1 source · 1 quote
MXFP8 block-scale regrouping at load used · 1 source · 1 quote
Native FP8 KV cache with offloading used · 1 source · 1 quote
NVFP4 post-training quantization used · 1 source · 1 quote
NVFP4 W4A4 quantization of MTP experts used · 1 source · 1 quote
NVIDIA ModelOpt quantization used · 1 source · 1 quote
OCP MXFP4 quantization used · 1 source · 1 quote
Parameter alignment for quantization used · 1 source · 1 quote
Platform-dependent W4A4/W4A16 quantization used · 1 source · 1 quote
Post-training calibration sample count used · 1 source · 1 quote
Quantization-aware distillation used · 1 source · 1 quote
QuaRot used · 1 source · 1 quote
Random Hadamard transform used · 1 source · 1 quote
SFT-mixture calibration used · 1 source · 1 quote
SpinQuant R1 rotation preprocessing used · 1 source · 1 quote
Static activation quantization used · 1 source · 1 quote
Targeted 2-bit quantization used · 1 source · 1 quote
W4A16 quantization used · 1 source · 1 quote
W4A8 quantization used · 1 source · 1 quote
Fine-grained FP8 quantization optional · 1 source · 1 quote
IQ4_XS optional · 1 source · 1 quote
NVFP4 quantization of MoE expert linears optional · 1 source · 1 quote
NVFP4 weight-and-activation quantization optional · 1 source · 1 quote
Q3_K_L optional · 1 source · 1 quote
Q4_0 optional · 1 source · 1 quote
Q4_K_S optional · 1 source · 1 quote
Separate target and MTP lm_head precision optional · 1 source · 1 quote
4-bit quantization evaluated · 1 source · 1 quote
8-bit and 4-bit quantization evaluated · 1 source · 1 quote
8-bit quantization evaluated · 1 source · 1 quote
BF16 expert activations evaluated · 1 source · 1 quote
Block-scaled NVFP4 KV-cache quantization evaluated · 1 source · 1 quote
Empirical bits-per-element budget selection evaluated · 1 source · 1 quote
FP8 or NVFP4 KV-cache quantization evaluated · 1 source · 1 quote
Linear E2M1 residual correction evaluated · 1 source · 1 quote
Promote expert weights from NVFP4 to FP8 evaluated · 1 source · 2 quotes
Quantized and reduced-precision inference evaluated · 1 source · 1 quote
Restricted-search MSE weight-scale selection evaluated · 1 source · 1 quote
Sensitivity-driven mixed-precision quantization evaluated · 1 source · 1 quote
Low-bit quantization mentioned · 1 source · 1 quote
Rotation-based quantization mitigation mentioned · 1 source · 1 quote
Mixed-precision key/value cache quantization not used · 1 source · 1 quote
On-the-fly block-wise FP8 weight quantization not used · 1 source · 1 quote
W4A4 quantization not used · 1 source · 1 quote

serving parallelism 20

Tensor parallelism default · 4 sources · 4 quotes
Attention data parallelism used · 5 sources · 6 quotes
DeepEP used · 4 sources · 4 quotes
Prefill-decode disaggregation used · 4 sources · 4 quotes
Topology-aware NVLink domain placement used · 2 sources · 2 quotes
Disaggregated serving optional · 2 sources · 2 quotes
14 single mentions
KDA state and KV-cache layout pinning core · 1 source · 1 quote
Data-parallel vision encoding used · 1 source · 1 quote
Disaggregated serving across 4+4 GPUs used · 1 source · 1 quote
FP8 parameter quantization during transfer used · 1 source · 1 quote
Tensor and expert parallelism used · 1 source · 1 quote
Tensor parallelism for MoE layers used · 1 source · 1 quote
TensorRT-LLM all-reduce backend used · 1 source · 1 quote
Expert parallelism optional · 1 source · 1 quote
Low-precision MoE combine optional · 1 source · 1 quote
MoonEP token migration for load balancing unclear · 1 source · 1 quote

inference scheduling 35

Asynchronous inference and execution core · 7 sources · 9 quotes
KV-cache-aware routing used · 2 sources · 2 quotes
NUMA binding for GPU-local CPU affinity used · 2 sources · 2 quotes
Provider failover routing used · 2 sources · 2 quotes
Provider routing used · 2 sources · 2 quotes
Session affinity routing used · 2 sources · 2 quotes
29 single mentions
Cross-group cache-hit block pinning core · 1 source · 1 quote
Dual-cluster consistent-hash failover core · 1 source · 1 quote
Latency-sensitive execution class core · 1 source · 1 quote
Threshold-triggered inference core · 1 source · 1 quote
Action-chunk timestep alignment used · 1 source · 1 quote
Eager adaptive depth scheduling used · 1 source · 1 quote
Envoy proxy with custom orchestrator used · 1 source · 1 quote
Greedy feasible-rank placement used · 1 source · 1 quote
Host-side scheduling optimization used · 1 source · 1 quote
Maximum-sequence tuning used · 1 source · 1 quote
Metric-driven inference-fleet autoscaling used · 1 source · 1 quote
NVLink domains as Ray custom resources used · 1 source · 1 quote
Static-shape compilation used · 1 source · 1 quote
Continuous batching optional · 1 source · 1 quote
Exacto routing optional · 1 source · 1 quote
Concurrent request batching evaluated · 1 source · 1 quote
Load balancing across providers mentioned · 1 source · 1 quote

inference kernel 59

Batch-invariant deterministic kernels core · 2 sources · 3 quotes
CUDA Graph capture and replay used · 4 sources · 6 quotes
CUDA graph capture size reduction used · 2 sources · 2 quotes
FlashAttention used · 2 sources · 2 quotes
FlashAttention-2 used · 2 sources · 2 quotes
FlashAttention-3 used · 2 sources · 2 quotes
Kernel fusion used · 2 sources · 2 quotes
MoE-side chunking used · 2 sources · 2 quotes
TensorRT-LLM multi-head attention backend used · 2 sources · 2 quotes
50 from one source, or only mentioned as not used
Fused AttnRes merge and RMSNorm core · 1 source · 1 quote
GPU-kernel selection core · 1 source · 1 quote
Separate-stream shared-expert GEMM overlap core · 1 source · 1 quote
Breakable CUDA graphs default · 1 source · 1 quote
Sparse pinned-host offload default · 1 source · 1 quote
Dynamic load balancing used · 1 source · 1 quote
Exp-free TopK kernel used · 1 source · 1 quote
Expert-optimized Triton kernels used · 1 source · 1 quote
FA4 blockwise attention used · 1 source · 1 quote
FA4 sheared-bias attention kernel used · 1 source · 1 quote
FlashKDA used · 1 source · 1 quote
FlexAttention used · 1 source · 1 quote
Fused mHC kernels used · 1 source · 1 quote
Fused QSA attention and KL-loss kernel used · 1 source · 1 quote
GPU-kernel optimization used · 1 source · 1 quote
Hidden-dimension split Combine kernel used · 1 source · 1 quote
Host Codegen used · 1 source · 1 quote
KDA algorithm-system co-design used · 1 source · 1 quote
Marlin NVFP4 kernels used · 1 source · 1 quote
Mega-mHC used · 1 source · 1 quote
Operator fusion using torch.compile used · 1 source · 1 quote
Persistent kernel used · 1 source · 1 quote
ROCm AITER sparse MLA attention backend used · 1 source · 1 quote
Sparse Flash Attention used · 1 source · 1 quote
Split-K CuTe GEMM used · 1 source · 1 quote
Triton MoE kernel used · 1 source · 1 quote
Two-phase forward used · 1 source · 1 quote
FlashAttention-4 optional · 1 source · 1 quote
HPC-Ops attention backend optional · 1 source · 1 quote
HPC-Ops fused MoE backend optional · 1 source · 1 quote
Multi-wave KV-cache splitting optional · 1 source · 1 quote
Per-quantization-scheme GEMM backend selection optional · 1 source · 1 quote
Persistent FlyDSL fused MoE decode kernel optional · 1 source · 1 quote
FP8 GEMM evaluated · 1 source · 1 quote
Avoiding split-K for batch invariance not used · 1 source · 1 quote

context management 24

Preserved thinking history core · 9 sources · 9 quotes
Excluding prior thinking from multi-turn history default · 6 sources · 6 quotes
Discard-all context management used · 6 sources · 7 quotes
Context compaction used · 3 sources · 3 quotes
Context folding used · 2 sources · 2 quotes
Context management method used · 2 sources · 2 quotes
Discard-75% evaluated · 2 sources · 2 quotes
Trajectory summarization and rollout re-initiation evaluated · 2 sources · 2 quotes
15 single mentions
Context reuse core · 1 source · 1 quote
Remove prior-turn reasoning unless preserved default · 1 source · 1 quote
Automatic prompt truncation used · 1 source · 1 quote
Cosine similarity used · 1 source · 1 quote
Discarding tool-call history used · 1 source · 1 quote
Four-step observation history used · 1 source · 1 quote
Hierarchical context management used · 1 source · 1 quote
Keep-recent-k used · 1 source · 1 quote
Memory compression used · 1 source · 1 quote
Inference-time vision-token budget scaling optional · 1 source · 1 quote
Modality-specific deployment optional · 1 source · 1 quote
Summary-based context compression evaluated · 1 source · 1 quote

agentic scaffolding 108

Function calling core · 3 sources · 3 quotes
Harmony format core · 3 sources · 3 quotes
Role-based instruction hierarchy core · 2 sources · 2 quotes
Automatic tool choice used · 4 sources · 4 quotes
Browser tool used · 3 sources · 4 quotes
Python tool for visual tasks used · 3 sources · 3 quotes
Robot control through tool orchestration used · 2 sources · 2 quotes
Thinking with tools used · 2 sources · 3 quotes
Tool-call parser used · 2 sources · 2 quotes
Visual Search Tool used · 2 sources · 2 quotes
XML-based tool-call schema with DSML token used · 2 sources · 2 quotes
Advisor strategy optional · 2 sources · 4 quotes
MCP configuration for tool definition optional · 2 sources · 2 quotes
Tool calling with the qwen3_coder parser optional · 2 sources · 2 quotes
93 single mentions
Agent Team mode core · 1 source · 6 quotes
Assistant output channels core · 1 source · 2 quotes
Chain-of-thought in the analysis channel core · 1 source · 1 quote
Developer message format core · 1 source · 1 quote
Harmony function-calling format core · 1 source · 1 quote
Harmony tool-call message format core · 1 source · 1 quote
Kolibri 1 reasoning parser core · 1 source · 1 quote
Periodic sandbox snapshots core · 1 source · 1 quote
Strict JSON Schema output core · 1 source · 1 quote
Visual trace steering core · 1 source · 2 quotes
Agentic search used · 1 source · 1 quote
Application-managed tool execution used · 1 source · 1 quote
Bash commands for context retrieval used · 1 source · 1 quote
Bash computer-use agent used · 1 source · 1 quote
Commentary-channel preambles used · 1 source · 1 quote
Concurrent subagent orchestration used · 1 source · 1 quote
Current-turn-only reasoning-mode detection used · 1 source · 1 quote
Deferred tool loading used · 1 source · 2 quotes
Developer-defined function schemas used · 1 source · 1 quote
Dynamic tool loading used · 1 source · 1 quote
Egress allowlist and prompt hardening used · 1 source · 1 quote
Embodiment-aware language prompts used · 1 source · 2 quotes
Harmony history end-token normalization used · 1 source · 1 quote
Harmony system message format used · 1 source · 1 quote
hy_v4 tool-call parser used · 1 source · 1 quote
Indexed parallel tool calls used · 1 source · 1 quote
Interactive natural-language steering used · 1 source · 1 quote
Interleaved tool calls in chain-of-thought used · 1 source · 1 quote
JSON Schema response formats used · 1 source · 1 quote
Jupyter Notebook code interpreter used · 1 source · 1 quote
Matching training and inference prefixes used · 1 source · 1 quote
MicroVM-based sandbox isolation used · 1 source · 1 quote
Minimal tool-set design used · 1 source · 1 quote
Multi-token streaming-delta handling used · 1 source · 1 quote
Prompt-enforced tool-call format used · 1 source · 1 quote
Python execution tool in chain-of-thought used · 1 source · 1 quote
Qwen3 XML tool-call parser used · 1 source · 1 quote
ReAct Toolbelt used · 1 source · 1 quote
Request timeouts and retries used · 1 source · 1 quote
Retrieval-Augmented Search used · 1 source · 1 quote
Runtime command filtering used · 1 source · 1 quote
Sandbox infrastructure (DSec) used · 1 source · 1 quote
Sandboxed Python execution loop used · 1 source · 1 quote
Search-efficiency constraints used · 1 source · 1 quote
Stateful Python tool used · 1 source · 1 quote
step3p5 tool-call parser used · 1 source · 1 quote
System-prompt guidance for tool use used · 1 source · 1 quote
Template-based tool formatting used · 1 source · 1 quote
Tool message format used · 1 source · 1 quote
Tool-Integrated Reasoning used · 1 source · 1 quote
Turn-end hook used · 1 source · 1 quote
Typed tool arguments used · 1 source · 1 quote
TypeScript-like function schema syntax used · 1 source · 1 quote
Unprivileged-user sandboxes used · 1 source · 1 quote
Vision in the loop used · 1 source · 1 quote
XML-tagged tool-call format used · 1 source · 1 quote
XTML chat template used · 1 source · 1 quote
Application-level output validation optional · 1 source · 1 quote
Checkpointing for rollback optional · 1 source · 1 quote
Claude Code skills for Tinker optional · 1 source · 1 quote
Compact TXT image-path notation optional · 1 source · 1 quote
Hermes-style tool calling optional · 1 source · 1 quote
Independent agent review gate optional · 1 source · 1 quote
JSON mode optional · 1 source · 1 quote
OpenAI-style JSON content blocks optional · 1 source · 1 quote
Retrieval or search optional · 1 source · 1 quote
StreamableParser optional · 1 source · 1 quote
Trajectory editing optional · 1 source · 1 quote
Video understanding and editing optional · 1 source · 1 quote
Vision-driven UI coding optional · 1 source · 1 quote
GPT-4 high-level policy evaluated · 1 source · 1 quote
Agentic tool calling mentioned · 1 source · 1 quote
Multi-agent collaboration mentioned · 1 source · 1 quote
native function calling mentioned · 1 source · 1 quote
Task-specification prompting mentioned · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelfiled heredecoding strategyreasoning controlKV cache managementinference quantizationserving parallelisminference schedulinginference kernelcontext managementagentic scaffolding
DeepSeek-V4.1-Flash-1017 used———Compressed KV caching used———Provider failover routing used———API request conversion, prompt encoding, and response parsing used—
GLM-5.3-Flash core—Multi-Token Prediction used—Always-on thinking mode coreChat-template reasoning-effort resolution and injection defaultThinking mode defaultclear_thinking chat-template flag optional——FP8 defaultMXFP8 block-scale regrouping at load usedFP8 KV-cache quantization optional—KDA state and KV-cache layout pinning coreDisaggregated serving across encoding, prefill, and decode usedPrefill-decode disaggregation usedRound-robin routing for prefill-decode disaggregation used——ROCm AITER sparse MLA attention backend used——Video understanding and editing optionalVision-driven UI coding optional—
DeepSeek-V4.1-Flash core—Confidence-scheduled verification coreDSpark coreSemi-autoregressive draft generation core—Capped linear reasoning-token length deduction coreContinuously controllable reasoning effort coreEffort-dependent exponential token-penalty schedule corePreset effort tiers mapped to scalar values defaultSystem-prompt reasoning-effort conditioning used—Compressed KV caching coreSWA Bounded Replay core—FP4 global KV-cache quantization core——Independent replica placement using predicted resource availability coreLatency-sensitive execution class coreNode-local placement validation with hard admission control coreSCHED_IDLE prioritization for non-latency-sensitive tasks coreSub-NUMA partitioning with NUMA-local worker VMs core—Mega-mHC used——Agent Team mode core—
Hy4-preview defaultCommunication optimization used—Multi-Token Prediction usedNEXTN speculative decoding usedSpeculative decoding used—Chain-of-thought reasoning defaultThinking mode used——FP8 usedLarge-model quantization and speculative sampling optional—Tensor parallelism used——Kernel fusion used——Automatic tool choice usedhy_v4 tool-call parser used—
Step-5-Preview coreOffline generation optional—Multi-layer EAGLE usedSpeculative decoding usedEAGLE optionalMulti-Token Prediction optional——Prefix caching optionalHiCache mentioned—FP8 optional—Attention data parallelism mentioned————Iterative tool execution with verification and revision core—
MiniMax-M3.1-Flash used—DSpark usedDSpark draft head optionalTemperature and nucleus sampling configuration optional—Thinking mode optional————Provider routing usedLoad balancing across providers mentioned————
GLM-5.3 default—Multi-Token Prediction used—clear_thinking chat-template flag defaultThinking mode default—Disabling prefix caching for benchmarking used—FP8 defaultFP8 KV-cache quantization usedNVFP4 quantization of MoE expert linears optional——Maximum-sequence tuning used——Context management method used——
DeepSeek-V4-Flash-0731 used—DSpark usedChat Prefix Completion optionalSpeculative decoding mentioned—Thinking mode used———————Function calling optionalJSON mode optional—
Hy3 core—Speculative decoding coreEAGLE optionalSpeculative sampling optional—Thinking mode defaultReasoning parser usedChain-of-thought reasoning optional——FP8 usedFP8 KV-cache quantization optionalQuantization optionalLow-bit quantization mentioned—TensorRT-LLM all-reduce backend used——Triton MoE kernel usedHPC-Ops attention backend optionalHPC-Ops fused MoE backend optional——Tool-call parser used—
MiMo-V2.6-Flash core—DFlash coreFive-layer SWA MTP drafter coreDistribution-matched speculative-draft fine-tuning usedThroughput-based draft-block sizing usedEAGLE optionalSpeculative decoding optional——Persistent per-dialogue-context KV caching coreAsynchronous KV-cache offloading usedHierarchical KV caching across HBM and pinned host memory used—Dynamic activation scaling coreFP8 low-precision speculative draft computation used—DeepEP used—Greedy feasible-rank placement used———Minimal, modular mini-harnesses for controlled harness diversity usedRequest timeouts and retries usedMulti-agent collaboration mentioned—
NVIDIA-Nemotron-3-Ultra-550B-A55B core—Recursive shared MTP-head drafting coreSpeculative decoding defaultBest-of-N selection usedEAGLE usedMulti-Token Prediction used—Chain-of-thought reasoning usedGenerate-verify-refine usedThinking mode usedTurn budget capping usedTurn-aware prompting usedReasoning token-budget control optionaleffort parameters mentioned—Chunked prefill defaultPrefix caching used8-bit Mamba cache quantization evaluatedPeriodic cache checkpointing evaluated—Mixed-FP8 layers in an NVFP4 recipe coreNVFP4 routed-expert GEMM quantization corePer-tensor FP8 quantization coreSelective BF16 retention for sensitive layers coreNVFP4 KV-cache quantization defaultFour-Over-Six usedFP16 cache storage with stochastic rounding usedFP8 usedFP8 KV-cache quantization usedHeuristic mixed per-layer precision quantization usedMamba SSM-cache quantization usedPost-training quantization usedRandom Hadamard transform usedW4A16 quantization usedBlock-scaled INT8 quantization with stochastic rounding evaluatedEmpirical bits-per-element budget selection evaluatedFP8 E4M3 quantization evaluatedMax-based weight scaling evaluatedMSE-based weight-scale selection evaluated—Prefill-decode disaggregation usedTopology-aware NVLink domain placement usedAttention data parallelism optionalLow-precision MoE combine optional—Composite-key topology-aware GPU rank assignment usedNUMA binding for GPU-local CPU affinity usedNVLink domains as Ray custom resources used—Marlin NVFP4 kernels usedMoE-side chunking used—Discard-all context management usedSummary-based context compression evaluated—Runtime command filtering usedTool-Integrated Reasoning used—
DeepSeek-V4-Flash core——Persistent reasoning history usedThink-tag response formatting usedThinking mode optional—Customized heterogeneous KV-cache layout coreDynamic allocation of fixed-size state-cache pools usedKV-cache and sparse-attention-kernel co-design usedOn-disk KV-cache storage for shared-prefix reuse usedPeriodic cache checkpointing optionalZero SWA caching optional—FP4 attention-indexer precision usedFP8 precision used———Batch-invariant deterministic kernels coreDeepGEMM-based batch-invariant matrix multiplication usedDistributed shared memory for cross-SM attention exchange usedDual-kernel batch-invariant attention decoding usedFused mHC kernels usedHost Codegen usedPer-SM accumulation buffers with deterministic global summation usedSeparate split-K outputs with deterministic reduction usedToken-order preprocessing and cross-rank buffer isolation usedAvoiding split-K for batch invariance not used——Agentic search usedRetrieval-Augmented Search usedXML-based tool-call schema with DSML token used—
DeepSeek-V3.2 used—Longest-thinking-trace selection usedMulti-stage candidate filtering usedTop-p and top-k sampling used—Generate-verify-refine usedReasoning-enabled boolean toggle optionalParallel test-time compute scaling evaluatedParallel-fewest-step sampling evaluatedSerial test-time compute scaling through context management evaluatedThinking mode evaluated——————Context management method usedTest-time context management for extending token budgets usedTool-use reasoning-context persistence and selective deletion usedDiscard-75% evaluatedDiscard-all context management evaluatedTrajectory summarization and rollout re-initiation evaluated—Cold-start prompting for reasoning in tool use usedJupyter Notebook code interpreter usedPrompt-enforced tool-call format usedThinking with tools used—
DeepSeek-V4-Flash-Vision-Exp default—Same-checkpoint target and draft weights defaultChat Prefix Completion optionalDSpark optional—Thinking mode optional——FP8 KV-cache quantization optional—————Compact TXT image-path notation optionalFunction calling optionalJSON mode optionalOpenAI-style JSON content blocks optional—
DeepSeek-V4-Pro core——Persistent reasoning history usedThink-tag response formatting usedThinking mode used—Customized heterogeneous KV-cache layout coreDynamic allocation of fixed-size state-cache pools usedKV-cache and sparse-attention-kernel co-design usedOn-disk KV-cache storage for shared-prefix reuse usedPeriodic cache checkpointing optionalZero SWA caching optional—FP4 attention-indexer precision usedFP8 precision used———Batch-invariant deterministic kernels coreDeepGEMM-based batch-invariant matrix multiplication usedDistributed shared memory for cross-SM attention exchange usedDual-kernel batch-invariant attention decoding usedFused mHC kernels usedHost Codegen usedPer-SM accumulation buffers with deterministic global summation usedSeparate split-K outputs with deterministic reduction usedToken-order preprocessing and cross-rank buffer isolation usedAvoiding split-K for batch invariance not used——Agentic search usedRetrieval-Augmented Search usedSandbox infrastructure (DSec) usedXML-based tool-call schema with DSML token used—
DeepSeek-V4-Pro-0813 optional—Chat Prefix Completion optionalDSpark optional—Thinking mode optional———————Function calling optionalJSON mode optional—
FLUX 3 Action usedPre-deployment robot-action safety checks used——————————
GLM-5.2 default——Thinking mode default—Prefix caching usedRadixCache used—Flex_AWQ_SSZ usedFP8 usedQuaRot usedW4A8 quantization used—Attention data parallelism usedPrefill-decode disaggregation used—Asynchronous inference and execution used—Sparse Flash Attention used—Discard-all context management usedHierarchical context management usedKeep-recent-k used——
GR00T N1.7 optionalServer-client policy inference optional——————————
Gemma 4 31B core—Multi-Token Prediction coreStandardized sampling configuration defaultClustered top-K projection usedDrafter-target KV-cache sharing used—control-token-enabled thinking mode coreThinking mode coreConfigurable thinking mode controlled by the <|think|> token optionalconfigurable thinking/reasoning mode for LLM inference optionalStep-by-step thinking mode optional—Key-as-value reuse in global attention layers usedKV cache sharing usedPrompt caching usedStateless in-memory prefix caching used—Mobile quantization coreQuantization coreChannel-wise quantization usedEmbedding and KV-cache quantization usedStatic activation quantization usedTargeted 2-bit quantization usedQ4_0 optionalPost-training quantization not used——Exacto routing optionalConcurrent request batching evaluated——Excluding prior thinking from multi-turn history defaultModality-specific deployment optional—Function calling coreSandboxed Python execution loop usednative function calling mentioned—
Inkling core—Multi-Token Prediction core—Continuously controllable reasoning effort usedThinking mode used—Prompt caching used—Mixed-precision NVFP4 routed-expert quantization default—Fused reduce-scatter/all-gather collectives used——FA4 sheared-bias attention kernel used——Python tool for visual tasks usedClaude Code skills for Tinker optional—
Kimi K3 core—Fused recurrent replay kernel coreMTP-to-EAGLE-3 draft-model fine-tuning used—Always-on thinking mode coreChain-of-thought reasoning defaultconfigurable reasoning effort via reasoning_effort (low/high/max) defaultThinking mode defaultGeneration-prefix mode selection usedNatural-language reasoning-effort option message usedReasoning token-budget control usedStage-wise effort-budget curriculum used—Fine-grained prefix hashing within cache pages coreKDA-aware prefix-cache management coreProjected-input caching for speculative KDA rollback coreShared-free-list cache allocation coreSparse hash-aligned KDA recurrent-state checkpoints coreUnified paged cache layout for KDA states and MLA KV coreWrite-back external KV-cache policy coreKDA with prefill cache usedKV-cache-aware placement of one-shot option messages used—Offline weight-layout permutation coreMXFP4 weight quantization usedMXFP8 activation quantization used—Zero-copy fused token permutation and unpermutation coreMoonEP token migration for load balancing unclear—Cross-group cache-hit block pinning coreDual-cluster consistent-hash failover corePrefix-cache-aware session affinity scheduling coreRequest-class resource-budget admission control coreRuntime-signal-based rollout concurrency auto-throttling coreWorkload-aware routed-expert GEMM scheduling core—Fused AttnRes merge and RMSNorm coreFused latent down-projection and MoE-router GEMM coreGPU kernel for online expert placement planning coreSeparate-stream shared-expert GEMM overlap coreSide-stream overlap for inter-block attention-residual computation coreSynchronization-free static-shape MoE execution coreWarpDecode token-centric routed-expert decoding coreFlashKDA usedKDA algorithm-system co-design used—Preserved thinking history coreContext compaction used—Function calling coreNatural-language instructions for request options corePeriodic sandbox snapshots coreConcurrent subagent orchestration usedCopy-on-write memory with page-cache optimization usedDynamic tool loading usedIndexed parallel tool calls usedmicroVM-isolated sandbox runtime for agentic AI usedOverlayBD shared-image sandbox launch stack usedTyped tool arguments usedVision in the loop usedXTML chat template usedAgentic tool calling mentioned—
Kolibri-1 core—Greedy decoding used—Thinking mode coreChat-template thinking-mode switch defaultSystem-prompt reasoning-effort conditioning defaultLength-dependent completion-budget signal usedSoft probabilistic assignment of reasoning-effort labels used———FP8 parameter quantization during transfer used—Priority-based, topology-aware GPU job scheduling used———Kolibri 1 reasoning parser coreMulti-step retrieval with grounded answering and abstention coreEgress allowlist and prompt hardening usedSearch-efficiency constraints usedUnprivileged-user sandboxes usedApplication-level output validation optionalHermes-style tool calling optionalIndependent agent review gate optional—
LFM3-1.7B defaultInterleaved text-and-audio generation optional—Greedy decoding usedTask-specific sampling-parameter presets optional—Reasoning enabled by default default——————Remove prior-turn reasoning unless preserved defaultAutomatic prompt truncation usedInference-time vision-token budget scaling optional—Application-managed tool execution usedDeferred tool loading usedRobot control through tool orchestration usedTemplate-based tool formatting usedRetrieval or search optional—
Laguna-S-2.1 core—DFlash optional—Interleaved thinking coreMax thinking mode with self-set compute budget defaultThinking mode defaultPersistent reasoning history usedPer-request thinking control optionalReasoning effort configuration availability not used—KV-cache reset on weight synchronization used—AWQ INT4 weight-only quantization usedFP8 KV-cache quantization usedFP8 KV-cache quantization with amax-based scaling usedFP8 W8A8 quantization with dynamic activation scaling usedLayer-wise mixed INT4/INT8 quantization usedMoE-layer quantization usedSpinQuant R1 rotation preprocessing usedTRT-LLM inference optimization to NVFP4 on Blackwell usedOn-the-fly block-wise FP8 weight quantization not used——Envoy proxy with custom orchestrator used——Preserved thinking history used—Current-turn-only reasoning-mode detection usedMulti-token streaming-delta handling usedXML-tagged tool-call format usedCheckpointing for rollback optionalTask-specification prompting mentioned—
LingBot-VLA 2.0 used———————FlexAttention usedOperator fusion using torch.compile used———
MiMo-V2.5 used—Multi-Token Prediction usedSpeculative decoding usedEAGLE optional——Chunked prefill usedRequest-level prefix cache usedRadixCache not used—FP8 used—DeepEP usedAttention data parallelism optional——FlashAttention-3 used—Discarding tool-call history usedMemory compression used—Bash commands for context retrieval used—
MiMo-V2.5-Pro used—Multi-Token Prediction usedEAGLE optionalMulti-layer EAGLE optional————DeepEP optional—————
MiMo-V2.6-Pro core—DFlash coreFive-layer SWA MTP drafter coreDistribution-matched speculative-draft fine-tuning usedThroughput-based draft-block sizing used—Chain-of-thought reasoning used—Persistent per-dialogue-context KV caching coreAsynchronous KV-cache offloading usedHierarchical KV caching across HBM and pinned host memory used—Dynamic activation scaling coreFP8 low-precision speculative draft computation used—DeepEP used—Greedy feasible-rank placement used———Minimal, modular mini-harnesses for controlled harness diversity usedRequest timeouts and retries used—
MiniMax-M3 core——Test-time compute scaling usedAdaptive reasoning optionalThinking mode optional—Block-based KV cache coreAutomatic cache support defaultPre-scheduled tile chunking used———Host-side scheduling optimization used—CUDA Graph capture and replay usedDynamic load balancing usedExp-free TopK kernel usedFlashAttention usedPersistent kernel usedTwo-phase forward usedFP8 GEMM evaluated——Producer–Verifier adversarial harness loop usedReAct Toolbelt used—
Mistral-Large-4 core—DSpark coreRecurrent Markov draft head coreThree-layer dense MLA speculative drafter coreThree-token speculative draft depth coreBlock verification usedRecurrent conditioning head usedTemperature and nucleus sampling configuration usedUnique mask tokens used—Reasoning cannot be disabled defaultThinking mode default—Cross-layer KV-cache sharing corePrefix caching defaultYou Only Cache Once (YOCO) usedDecode context parallelism optional—FP8 latent KV-cache quantization defaultAbs-max quantization usedFP8 usedMixed-precision NVFP4 routed-expert quantization usedNative FP8 KV cache with offloading usedOCP MXFP4 quantization usedSFT-mixture calibration usedLayer-by-layer FP8-to-MXFP4 expert re-quantization at load optional—Disaggregated serving across 4+4 GPUs usedTensor and expert parallelism used—Metric-driven inference-fleet autoscaling usedSession-sticky routing to data-parallel ranks usedProvider routing mentioned—Breakable CUDA graphs defaultFA4 blockwise attention usedKernel fusion usedPersistent FlyDSL fused MoE decode kernel optionalSingle-launch fusion of attention, indexer, routing, experts, and all-reduces optional——Strict JSON Schema output coreMicroVM-based sandbox isolation usedMinimal tool-set design usedSystem-prompt guidance for tool use usedTurn-end hook used—
MolmoAct 2 corePre-deployment robot-action safety checks optionalReusable-cache optimization optional—Action-chunk execution usedAutoregressive discrete action-token inference optional—Adaptive reasoning core—Cached depth-token replay for unchanged cells usedPreallocated static KV cache used———Caching invariant action-expert intermediates usedEager adaptive depth scheduling usedStatic-shape compilation used—CUDA Graph capture and replay used—Adaptive depth-token prediction with cached unchanged cells coreCosine similarity used—Visual trace steering coreInteractive natural-language steering usedMatching training and inference prefixes usedPrompt descriptors for embodiment and control semantics usedPrompting Molmo to locate the robot gripper usedTest-time action steering through language or visual traces optionalTrajectory editing optional—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B corePlatform- and workload-specific quantization and speculative-decoding tuning used—Speculative decoding coreTop-K candidate lattice coreBest-of-N selection usedLiLiCorr single-pass correlation head usedMulti-Token Prediction usedSemi-autoregressive drafting with sequential Markov correction usedStandalone parallel speculative drafters usedDFlash optionalDSpark optionalProbabilistic draft sampling optionalConcurrency-aware draft-length tuning evaluatedConfidence head not used—Reasoning token-budget control usedParallel test-time compute scaling evaluated—Checkpointing replay for Mamba recurrent-state cache usedHybrid-state caching usedInternal prefill checkpointing usedMamba checkpointing for speculative-decoding replay optional—Backbone-coupled MTP KV-cache precision coreShared FP8 lm_head for target verification and MTP drafting defaultAggressive mixed-precision quantization recipe usedBF16 MTP attention weights with FP8 K/V BMM quantizers usedFP8-or-higher lm_head precision usedGradient-based per-layer mixed-precision quantization search usedMSE-based weight-scale selection usedNVFP4 post-training quantization usedNVFP4 W4A4 quantization of MTP experts usedNVFP4 weight quantization with BF16 activations usedParameter alignment for quantization usedPer-block FP8 quantization for attention Q/K/V projections usedPost-training calibration sample count usedPost-training quantization usedQuantization-aware distillation usedSeparate MTP quantization and backbone splicing usedTask-loss-gradient-weighted quantization sensitivity estimation usedWeight-only post-training quantization for parallel drafters usedNVFP4 quantization optionalSeparate target and MTP lm_head precision optionalBF16 expert activations evaluatedBlock-scaled NVFP4 KV-cache quantization evaluatedFP4 quantization of Mamba output-projection weights evaluatedFP8 or NVFP4 KV-cache quantization evaluatedIncrease MoE expert precision and fine-tune on RL rollouts evaluatedLinear E2M1 residual correction evaluatedMXFP8 block-scaled attention projection quantization evaluatedPer-tensor FP8 quantization evaluatedPromote expert weights from NVFP4 to FP8 evaluatedQuantization of the parallel-drafter language-model head evaluatedRestricted-search MSE weight-scale selection evaluatedRotation or higher precision for Mamba input activations evaluatedSensitivity-driven mixed-precision quantization evaluatedTwo-pass FP4 weight quantization with residual correction evaluatedW4A16 quantization of Mamba output-projection weights evaluatedHigher-precision activations for Mamba output projection mentionedRotation-based quantization mitigation mentionedMixed-precision key/value cache quantization not usedW4A4 quantization not used—Disaggregated serving evaluated—KV-cache-aware routing usedSession affinity routing used—GPU-kernel optimization usedMulti-wave KV-cache splitting optionalPer-quantization-scheme GEMM backend selection optional——Model routing by planning and execution role coreBash computer-use agent usedParallel agent execution with self-verification used—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B corePlatform- and workload-specific quantization and speculative-decoding tuning used—Speculative decoding coreTop-K candidate lattice coreLiLiCorr single-pass correlation head usedSemi-autoregressive drafting with sequential Markov correction usedStandalone parallel speculative drafters usedDFlash optionalDSpark optionalLiLiCorr optionalMTP draft-length configuration optionalProbabilistic draft sampling optionalConfidence head not used—Reasoning enabled by default defaultReasoning token-budget control optionalThinking mode optionalParallel test-time compute scaling evaluated—Attention-cache and Mamba-state preservation coreHybrid-state caching corePrefix caching corePrefix reuse coreCheckpointing replay for Mamba recurrent-state cache usedInternal prefill checkpointing usedMamba checkpointing for speculative-decoding replay usedKV-cache management optionalKV-cache offloading evaluated—Backbone-coupled MTP KV-cache precision coreMixed-precision inference coreNative 2-bit quantization coreShared FP8 lm_head for target verification and MTP drafting defaultAggressive mixed-precision quantization recipe usedBF16 MTP attention weights with FP8 K/V BMM quantizers usedComponent-wise precision selection usedFP8-or-higher lm_head precision usedGradient-based per-layer mixed-precision quantization search usedMSE-based weight-scale selection usedNVFP4 post-training quantization usedNVFP4 W4A4 quantization of MTP experts usedNVFP4 weight quantization with BF16 activations usedParameter alignment for quantization usedPer-block FP8 quantization for attention Q/K/V projections usedPlatform-dependent W4A4/W4A16 quantization usedPost-training calibration sample count usedPost-training quantization usedQuantization-aware distillation usedSeparate MTP quantization and backbone splicing usedTask-loss-gradient-weighted quantization sensitivity estimation usedWeight-only post-training quantization for parallel drafters usedNVFP4 weight-and-activation quantization optionalSeparate target and MTP lm_head precision optionalBF16 expert activations evaluatedBlock-scaled NVFP4 KV-cache quantization evaluatedFP4 quantization of Mamba output-projection weights evaluatedFP8 or NVFP4 KV-cache quantization evaluatedIncrease MoE expert precision and fine-tune on RL rollouts evaluatedLinear E2M1 residual correction evaluatedMXFP8 block-scaled attention projection quantization evaluatedPer-tensor FP8 quantization evaluatedPromote expert weights from NVFP4 to FP8 evaluatedQuantization of the parallel-drafter language-model head evaluatedRestricted-search MSE weight-scale selection evaluatedRotation or higher precision for Mamba input activations evaluatedSensitivity-driven mixed-precision quantization evaluatedTwo-pass FP4 weight quantization with residual correction evaluatedW4A16 quantization of Mamba output-projection weights evaluatedHigher-precision activations for Mamba output projection mentionedRotation-based quantization mitigation mentionedMixed-precision key/value cache quantization not usedW4A4 quantization not used—Disaggregated serving optional—KV-cache-aware routing usedSession affinity routing usedAsynchronous inference and execution optionalProvider failover routing optional—GPU-kernel selection coreGPU-kernel optimization usedMulti-wave KV-cache splitting optionalPer-quantization-scheme GEMM backend selection optional—Context reuse core—Parallel agent execution with self-verification used—
OpenVLA / OpenVLA-OFT defaultRemote VLA inference server used—Greedy decoding usedSecond-most-likely action querying usedSpeculative decoding mentionedDynamic action querying not used———BF16 inference defaultQuantization optional4-bit quantization evaluated8-bit and 4-bit quantization evaluated8-bit quantization evaluatedQuantized and reduced-precision inference evaluated———FlashAttention optionalFlashAttention-2 optional———
Qwen3.5-397B-A17B default—MTP-1 speculative decoding usedPresence penalty tuning usedMulti-Token Prediction optionalNEXTN speculative decoding optionalTask-specific sampling-parameter presets optional—Reasoning enabled by default defaultThinking mode defaultDisable reasoning via enable_thinking optionalTask- and mode-specific sampling parameters optionalTask-appropriate maximum output length optionalQwen3 soft thinking switch not used—Chunked prefill usedLanguage-model-only serving mode usedPrompt caching usedPrefix caching optional—FP8 usedNVFP4 quantization used—Data-parallel vision encoding usedTensor parallelism optional———Excluding prior thinking from multi-turn history defaultContext folding usedDiscard-all context management evaluated—Talker system prompt for voice characteristics usedMCP configuration for tool definition optionalTool calling with the qwen3_coder parser optional—
Qwen3.6-35B-A3B default—Multi-Token Prediction optionalTask-specific sampling-parameter presets optional—Interleaved thinking defaultThinking mode defaultPersistent reasoning history usedDisable reasoning via enable_thinking optional—Chunked prefill usedPrefix caching usedPrompt caching usedLanguage-model-only serving mode optionalMamba prefix caching in align mode evaluated—FP8 KV-cache quantization usedFP8 optionalModelOpt NVFP4 re-quantization optionalNVFP4 quantization optional——Asynchronous inference and execution used—CUDA graph capture size reduction used—Preserved thinking history used—Automatic tool choice usedQwen3 XML tool-call parser usedMCP configuration for tool definition optional—
Qwen3.8-Flash-Next core—NEXTN speculative decoding usedMulti-Token Prediction optional—Reasoning enabled by default default—Overlapped host-memory prefetching used—Fine-grained FP8 quantization optional—Tensor parallelism defaultExpert parallelism optional—Continuous batching optional—Shape-aware kernel dispatch for Hyper-Connection coreSparse pinned-host offload defaultCUDA Graph capture and replay usedFused QSA attention and KL-loss kernel usedFusing activation, gating, and reduction into GEMM epilogues usedHidden-dimension split Combine kernel usedSplit-K CuTe GEMM used—Preserved thinking history default——
SmolVLA coreChunk fusion core—Action-chunk execution evaluated—————Asynchronous inference and execution coreThreshold-triggered inference core————
Step-3.7-Flash used—EAGLE optionalMulti-layer EAGLE optional—Reasoning parser usedStep3p5 reasoning parser usedThinking mode optional——FP16 multimodal projector usedFP8 usedFP8 KV-cache quantization usedGGUF usedModelOpt FP4 quantization usedModelOpt NVFP4 re-quantization usedNVFP4 quantization usedNVIDIA ModelOpt quantization usedIQ4_XS optionalQ3_K_L optionalQ4_K_S optional——Asynchronous inference and execution used—TensorRT-LLM multi-head attention backend usedFlashAttention-4 optional——Automatic tool choice usedPython tool for visual tasks usedstep3p5 tool-call parser usedTool-call parser usedVisual Search Tool usedAdvisor strategy optional—
VLA Foundry (Foundry-VLA-1.7B) used————————Temporal action chunking with past-context conditioning used——
X-VLA used—————————Embodiment-aware language prompts used—
Xiaomi-Robotics-1 core——————Asynchronous inference and execution coreAction-chunk timestep alignment used—FlashAttention-2 used—Four-step observation history used——
gpt-oss-120b core—Grammar-constrained decoding usedBest-of-N selection evaluated—Full chain-of-thought output coreThinking mode used——BF16 inference usedMXFP4 block-scaled tensor packing usedMXFP4 weight quantization used—Tensor parallelism for MoE layers used——CUDA Graph capture and replay usedExpert-optimized Triton kernels usedOptimized Triton MoE kernel with MXFP4 support used—Excluding prior thinking from multi-turn history used—Assistant output channels coreChain-of-thought in the analysis channel coreDeveloper message format coreHarmony format coreHarmony function-calling format coreHarmony tool-call message format coreRole-based instruction hierarchy coreBrowser tool usedBuilt-in tool declaration in the system message usedCommentary-channel preambles usedDeveloper-defined function schemas usedHarmony history end-token normalization usedHarmony system message format usedInterleaved tool calls in chain-of-thought usedJSON Schema response formats usedPython execution tool in chain-of-thought usedRetaining chain-of-thought across tool calls usedStateful Python tool usedStateless Python tool reference implementation usedTool message format usedTypeScript-like function schema syntax usedStreamableParser optional—
π0.5 core—Greedy decoding defaultTemperature sampling for bimanual tasks optional—Hierarchical subtask inference coreHigh-level language policy for task decomposition used—Robot-state KV caching across sampling steps corePrefix key-value caching during action inference used——————GPT-4 high-level policy evaluated—

Proposed children

Paths the corpus wanted and the taxonomy does not have. A human promotes them into the outline; the extraction cannot.

provider routing and failovercross-provider routing, healthy-provider failover, provider fallback
↑