implementation detail · filed under model architecture
N-gram embedding host-memory offload and prefetch
N-gram embedding tables are stored in host memory and asynchronously prefetched or offloaded to reduce GPU-memory use while overlapping transfers with model computation.
Also called asynchronously offloaded to host memory, asynchronous prefetching, Host-memory offload for N-gram embeddings.
- sources
- 4
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Evidence
6 spans quoted from the sources, strongest treatment first.
Capacity is further added outside the backbone by a single n-gram embedding layer whose tables are prefetched from host memory
The N-gram Embedding can reside entirely in host memory to save GPU memory
lookup positions are computed in advance and asynchronously prefetched, so it never permanently occupies GPU memory.
can be asynchronously offloaded to host memory.
the embedding table can be offloaded to host memory
overlapped with model computation through asynchronous prefetching
Filed alongside
Other methods under model architecture :: context capacity.