Model techniques map
Taxonomyinference & servinginference kernel

taxonomy node · level 2

inference kernel

50 methods filed at this node or below it, from the sources of 16 models.

inference & serving :: inference kernel

Matching aids for the classifier: FlashAttention; Triton MoE kernel; CUTLASS; CUDA graph; persistent kernel; exp-free TopK kernel; batch-invariant deterministic kernels.

In this branch 50

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 50

Batch-invariant deterministic kernels core · 2 sources · 3 quotes
Kernel fusion core · 2 sources · 2 quotes
Fused AttnRes merge and RMSNorm kernel core · 1 source · 1 quote
Fused latent down-projection and MoE-router GEMM core · 1 source · 1 quote
GPU planning kernel for online expert placement core · 1 source · 1 quote
Separate-stream shared-expert GEMM overlap core · 1 source · 1 quote
Shape-aware kernel dispatch for Hyper-Connection core · 1 source · 1 quote
Side-stream overlap for inter-block attention-residual computation core · 1 source · 1 quote
Synchronization-free static-shape MoE execution core · 1 source · 2 quotes
WarpDecode token-centric routed-expert decoding kernel core · 1 source · 1 quote
Sparse pinned-host offload default · 1 source · 1 quote
CUDA Graph used · 3 sources · 3 quotes
CUDA Graph capture size reduction used · 2 sources · 2 quotes
FlashAttention 3 used · 2 sources · 2 quotes
MoE-side chunking used · 2 sources · 2 quotes
TensorRT-LLM multi-head attention backend used · 2 sources · 2 quotes
DeepGEMM-based batch-invariant matrix multiplication used · 1 source · 1 quote
Distributed shared memory for cross-SM attention data exchange used · 1 source · 1 quote
Dual-kernel batch-invariant attention decoding used · 1 source · 1 quote
Dynamic load balancing used · 1 source · 1 quote
Exp-free TopK kernel used · 1 source · 1 quote
Expert-optimized Triton kernels used · 1 source · 1 quote
FA4 sheared-bias attention kernel used · 1 source · 1 quote
FlashAttention used · 1 source · 1 quote
FlashKDA used · 1 source · 1 quote
Fused mHC kernels used · 1 source · 1 quote
Fused QSA kernel used · 1 source · 1 quote
Fused RoPE-attention-RoPE-cast kernel used · 1 source · 1 quote
Fusion of activation, gating, and reduction into GEMM epilogues used · 1 source · 1 quote
Hidden-dimension split Combine kernel used · 1 source · 1 quote
Host Codegen used · 1 source · 1 quote
KDA algorithm–system co-design used · 1 source · 1 quote
Marlin NVFP4 kernels used · 1 source · 1 quote
Mega-mHC used · 1 source · 1 quote
Optimized Triton MoE kernel with MXFP4 support used · 1 source · 1 quote
Per-SM accumulation buffers with deterministic global summation used · 1 source · 1 quote
Persistent kernel used · 1 source · 1 quote
ROCm AITER sparse MLA attention backend used · 1 source · 1 quote
Separate split-K outputs with deterministic reduction used · 1 source · 1 quote
Single-Pass mHC used · 1 source · 1 quote
Sparse Flash Attention used · 1 source · 1 quote
Split-K CuTe GEMM for low-batch Hyper-Connection Mix used · 1 source · 1 quote
Token-order preprocessing and cross-rank buffer isolation for deterministic MoE backward used · 1 source · 1 quote
Triton MoE kernel used · 1 source · 1 quote
Two-phase forward used · 1 source · 1 quote
FlashAttention 4 optional · 1 source · 1 quote
HPC-Ops attention backend optional · 1 source · 1 quote
HPC-Ops fused MoE backend optional · 1 source · 1 quote
FP8 GEMM evaluated · 1 source · 1 quote
Avoiding split-K not used · 1 source · 1 quote

By model

Which of this branch's techniques each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modeltechniques
GLM-5.3-Flash usedROCm AITER sparse MLA attention backend used—
DeepSeek-V4.1-Flash coreKernel fusion coreFused RoPE-attention-RoPE-cast kernel usedMega-mHC usedSingle-Pass mHC used—
Hy4-preview usedKernel fusion used—
NVIDIA-Nemotron-3-Ultra-550B-A55B usedMarlin NVFP4 kernels usedMoE-side chunking used—
DeepSeek-V4-Flash coreBatch-invariant deterministic kernels coreDeepGEMM-based batch-invariant matrix multiplication usedDistributed shared memory for cross-SM attention data exchange usedDual-kernel batch-invariant attention decoding usedFused mHC kernels usedHost Codegen usedPer-SM accumulation buffers with deterministic global summation usedSeparate split-K outputs with deterministic reduction usedToken-order preprocessing and cross-rank buffer isolation for deterministic MoE backward usedAvoiding split-K not used—
MiMo-V2.5 usedFlashAttention 3 used—
Hy3 usedTriton MoE kernel usedHPC-Ops attention backend optionalHPC-Ops fused MoE backend optional—
GLM-5.2 usedSparse Flash Attention used—
MiniMax-M3 usedCUDA Graph usedDynamic load balancing usedExp-free TopK kernel usedFlashAttention usedPersistent kernel usedTwo-phase forward usedFP8 GEMM evaluated—
DeepSeek-V4-Pro coreBatch-invariant deterministic kernels coreDeepGEMM-based batch-invariant matrix multiplication usedDistributed shared memory for cross-SM attention data exchange usedDual-kernel batch-invariant attention decoding usedFused mHC kernels usedHost Codegen usedPer-SM accumulation buffers with deterministic global summation usedSeparate split-K outputs with deterministic reduction usedToken-order preprocessing and cross-rank buffer isolation for deterministic MoE backward usedAvoiding split-K not used—
Inkling usedFA4 sheared-bias attention kernel used—
Kimi K3 coreFused AttnRes merge and RMSNorm kernel coreFused latent down-projection and MoE-router GEMM coreGPU planning kernel for online expert placement coreSeparate-stream shared-expert GEMM overlap coreSide-stream overlap for inter-block attention-residual computation coreSynchronization-free static-shape MoE execution coreWarpDecode token-centric routed-expert decoding kernel coreFlashKDA usedKDA algorithm–system co-design used—
Qwen3.6-35B-A3B usedCUDA Graph capture size reduction used—
Qwen3.8-Flash-Next coreShape-aware kernel dispatch for Hyper-Connection coreSparse pinned-host offload defaultCUDA Graph usedFused QSA kernel usedFusion of activation, gating, and reduction into GEMM epilogues usedHidden-dimension split Combine kernel usedSplit-K CuTe GEMM for low-batch Hyper-Connection Mix used—
Step-3.7-Flash usedTensorRT-LLM multi-head attention backend usedFlashAttention 4 optional—
gpt-oss-120b usedCUDA Graph usedExpert-optimized Triton kernels usedOptimized Triton MoE kernel with MXFP4 support used—