Open Models Almanac
Methodsinference & servinginference kernel

general family · filed under inference & serving

CUDA Graph capture and replay

CUDA Graph capture and replay records a fixed-shape computation as a graph and replays it, largely eliminating kernel launch overhead.1 It speeds up workloads with regular, repeated computation patterns, while sequential, variable-length execution is less amenable to graph capture.2,3

Trade-off It gives large speedups on regular fixed-shape workloads but smaller gains when execution is autoregressive and variable-length.4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. whose runtime is dominated by kernel launch overhead, which graph replay largely eliminates — MolmoAct2 Action Reasoning Models for Real-World Deployment
  2. its fixed-shape flow-matching steps form a regular, repeated computation pattern — MolmoAct2 Action Reasoning Models for Real-World Deployment
  3. whose sequential dependencies and variable-length execution are less amenable to graph capture — MolmoAct2 Action Reasoning Models for Real-World Deployment
  4. MolmoAct2-Think sees a smaller gain, since its adaptive-depth stage performs autoregressive decoding — MolmoAct2 Action Reasoning Models for Real-World Deployment

Also called CUDA Graph, CUDA Graph integration, the step is captured in a CUDA graph, CUDA Graph replay, CUDA graphs and basic caching.

sources
4
models
4
labs adopt it
4
strongest
used

Documented in

Alibaba (Qwen)
Allen Institute for AI
MiniMax

Evidence

6 quotes from 4 sources, one card per model, each opening with its strongest.

How sources treat it
optional 1used 5

MolmoAct 2

Allen Institute for AIadopted1 source

For regenerated depth tokens, we capture the fixed-shape transformer work from post-attention through the next layer’s pre-attention as CUDA Graph stages, and leave attention itself eager

usedinference servingin MolmoAct2-Think
2 more from this model

We further capture the fixed-shape flow loop with CUDA Graphs, reducing Python and kernel-launch overhead during repeated action generation.

usedinference servingin MolmoAct2

Enabling CUDA Graphs gives the larger gain

optionalunclearin MolmoAct2Allen Institute for Artificial Intelligence (AI2)

gpt-oss-120b

OpenAIadopted1 source

incl. using CUDA graphs and basic caching

usedsoftware implementationin gpt-oss reference Triton implementationOpenAI
↑