Open Models Almanac
Methodsinference & servinginference kernel

general family · filed under inference & serving

Kernel fusion

Kernel fusion merges a chain of GPU operations, such as a normalization and the rounding to FP8 that follows it, into one kernel, so the intermediate result stays in registers or shared memory instead of being written to and read back from HBM. It saves memory traffic and launches, and each fused kernel is written for one chain of operations, one layout and one format.

Written for the explainer Kernels and graph compilation: where the cost model meets the GPU.

Also called inference kernel fusion, systematic kernel fusing, operator fusion.

sources
2
models
2
labs adopt it
2
strongest
used
↑