single mention · inference & serving
Persistent kernel
A persistent kernel stays resident on the GPU and pulls work from a queue instead of being launched once per operation, which removes the idle gap between kernels that even a CUDA graph leaves. The extreme form, a megakernel, runs a whole forward pass as one kernel and is written for one model and one GPU.
Written for the explainer Kernels and graph compilation: where the cost model meets the GPU.
Also called persistent kernel rewriting.
Filed under inference & serving › inference kernel