Open Models Almanac
Methodsinference & servinginference kernel

methods · inference & serving

inference kernel

59 methods filed at this node or below it, from the sources of 23 models.

In this branch 59

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 59

Batch-invariant deterministic kernels core · 2 sources · 3 quotes
CUDA Graph capture and replay used · 4 sources · 6 quotes
CUDA graph capture size reduction used · 2 sources · 2 quotes
FlashAttention used · 2 sources · 2 quotes
FlashAttention-2 used · 2 sources · 2 quotes
FlashAttention-3 used · 2 sources · 2 quotes
Kernel fusion used · 2 sources · 2 quotes
MoE-side chunking used · 2 sources · 2 quotes
TensorRT-LLM multi-head attention backend used · 2 sources · 2 quotes
50 from one source, or only mentioned as not used
Fused AttnRes merge and RMSNorm core · 1 source · 1 quote
GPU-kernel selection core · 1 source · 1 quote
Separate-stream shared-expert GEMM overlap core · 1 source · 1 quote
Breakable CUDA graphs default · 1 source · 1 quote
Sparse pinned-host offload default · 1 source · 1 quote
Dynamic load balancing used · 1 source · 1 quote
Exp-free TopK kernel used · 1 source · 1 quote
Expert-optimized Triton kernels used · 1 source · 1 quote
FA4 blockwise attention used · 1 source · 1 quote
FA4 sheared-bias attention kernel used · 1 source · 1 quote
FlashKDA used · 1 source · 1 quote
FlexAttention used · 1 source · 1 quote
Fused mHC kernels used · 1 source · 1 quote
Fused QSA attention and KL-loss kernel used · 1 source · 1 quote
GPU-kernel optimization used · 1 source · 1 quote
Hidden-dimension split Combine kernel used · 1 source · 1 quote
Host Codegen used · 1 source · 1 quote
KDA algorithm-system co-design used · 1 source · 1 quote
Marlin NVFP4 kernels used · 1 source · 1 quote
Mega-mHC used · 1 source · 1 quote
Operator fusion using torch.compile used · 1 source · 1 quote
Persistent kernel used · 1 source · 1 quote
ROCm AITER sparse MLA attention backend used · 1 source · 1 quote
Sparse Flash Attention used · 1 source · 1 quote
Split-K CuTe GEMM used · 1 source · 1 quote
Triton MoE kernel used · 1 source · 1 quote
Two-phase forward used · 1 source · 1 quote
FlashAttention-4 optional · 1 source · 1 quote
HPC-Ops attention backend optional · 1 source · 1 quote
HPC-Ops fused MoE backend optional · 1 source · 1 quote
Multi-wave KV-cache splitting optional · 1 source · 1 quote
Per-quantization-scheme GEMM backend selection optional · 1 source · 1 quote
Persistent FlyDSL fused MoE decode kernel optional · 1 source · 1 quote
FP8 GEMM evaluated · 1 source · 1 quote
Avoiding split-K for batch invariance not used · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelmethods
GLM-5.3-Flash usedROCm AITER sparse MLA attention backend used—
DeepSeek-V4.1-Flash usedMega-mHC used—
Hy4-preview usedKernel fusion used—
Hy3 usedTriton MoE kernel usedHPC-Ops attention backend optionalHPC-Ops fused MoE backend optional—
NVIDIA-Nemotron-3-Ultra-550B-A55B usedMarlin NVFP4 kernels usedMoE-side chunking used—
DeepSeek-V4-Flash coreBatch-invariant deterministic kernels coreDeepGEMM-based batch-invariant matrix multiplication usedDistributed shared memory for cross-SM attention exchange usedDual-kernel batch-invariant attention decoding usedFused mHC kernels usedHost Codegen usedPer-SM accumulation buffers with deterministic global summation usedSeparate split-K outputs with deterministic reduction usedToken-order preprocessing and cross-rank buffer isolation usedAvoiding split-K for batch invariance not used—
DeepSeek-V4-Pro coreBatch-invariant deterministic kernels coreDeepGEMM-based batch-invariant matrix multiplication usedDistributed shared memory for cross-SM attention exchange usedDual-kernel batch-invariant attention decoding usedFused mHC kernels usedHost Codegen usedPer-SM accumulation buffers with deterministic global summation usedSeparate split-K outputs with deterministic reduction usedToken-order preprocessing and cross-rank buffer isolation usedAvoiding split-K for batch invariance not used—
GLM-5.2 usedSparse Flash Attention used—
Inkling usedFA4 sheared-bias attention kernel used—
Kimi K3 coreFused AttnRes merge and RMSNorm coreFused latent down-projection and MoE-router GEMM coreGPU kernel for online expert placement planning coreSeparate-stream shared-expert GEMM overlap coreSide-stream overlap for inter-block attention-residual computation coreSynchronization-free static-shape MoE execution coreWarpDecode token-centric routed-expert decoding coreFlashKDA usedKDA algorithm-system co-design used—
LingBot-VLA 2.0 usedFlexAttention usedOperator fusion using torch.compile used—
MiMo-V2.5 usedFlashAttention-3 used—
MiniMax-M3 usedCUDA Graph capture and replay usedDynamic load balancing usedExp-free TopK kernel usedFlashAttention usedPersistent kernel usedTwo-phase forward usedFP8 GEMM evaluated—
Mistral-Large-4 defaultBreakable CUDA graphs defaultFA4 blockwise attention usedKernel fusion usedPersistent FlyDSL fused MoE decode kernel optionalSingle-launch fusion of attention, indexer, routing, experts, and all-reduces optional—
MolmoAct 2 usedCUDA Graph capture and replay used—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B usedGPU-kernel optimization usedMulti-wave KV-cache splitting optionalPer-quantization-scheme GEMM backend selection optional—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B coreGPU-kernel selection coreGPU-kernel optimization usedMulti-wave KV-cache splitting optionalPer-quantization-scheme GEMM backend selection optional—
OpenVLA / OpenVLA-OFT optionalFlashAttention optionalFlashAttention-2 optional—
Qwen3.6-35B-A3B usedCUDA graph capture size reduction used—
Qwen3.8-Flash-Next coreShape-aware kernel dispatch for Hyper-Connection coreSparse pinned-host offload defaultCUDA Graph capture and replay usedFused QSA attention and KL-loss kernel usedFusing activation, gating, and reduction into GEMM epilogues usedHidden-dimension split Combine kernel usedSplit-K CuTe GEMM used—
Step-3.7-Flash usedTensorRT-LLM multi-head attention backend usedFlashAttention-4 optional—
Xiaomi-Robotics-1 usedFlashAttention-2 used—
gpt-oss-120b usedCUDA Graph capture and replay usedExpert-optimized Triton kernels usedOptimized Triton MoE kernel with MXFP4 support used—
↑