Open Models Almanac
Methodsmodel architecturecontext capacity

implementation detail · filed under model architecture

Host-memory offload for N-gram embeddings

Host-memory offload stores the N-gram embedding table in host memory instead of on the accelerator's memory.1 Accesses to the table are overlapped with model computation by asynchronously prefetching embeddings ahead of use.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. the embedding table can be offloaded to host memory — Qwen3.8-Flash-Next
  2. overlapped with model computation through asynchronous prefetching — Qwen3.8-Flash-Next

Also called asynchronously offloaded to host memory.

sources
3
model
1
lab adopt it
1
strongest
used

Documented in

Alibaba (Qwen)

Evidence

3 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
optional 2used 1

Qwen3.8-Flash-Next

Alibaba (Qwen)adopted3 sources

The N-gram Embedding can reside entirely in host memory to save GPU memory

usedunclearin Qwen3.8-Flash-NextRadixArk
2 more from this model

the embedding table can be offloaded to host memory

optionalinference servingin Qwen3.8-Flash-NextQwen Team

can be asynchronously offloaded to host memory.

optionalinference servingin Qwen3.8-Flash-NextQwen Team
↑