general family · filed under model architecture
N-gram embedding lookup memory
N-gram embedding lookup memory is an embedding-based memory that scales model capacity by retrieving stored vectors with short n-grams as keys.1 It generalizes unigram lookup by conditioning retrieval on local context rather than token identity alone, and injects the retrieved embeddings into the residual stream.2
Trade-off It scales capacity with negligible additional per-token FLOPs, and its deterministic addressing enables host-memory offloading and asynchronous prefetching.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
short n-grams ending at each token serve as keys into embedding tables, and the retrieved vectors augment the corresponding token representation
— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training StabilityN-gram embeddings further generalize unigram lookup by conditioning memory retrieval on local context rather than token identity alone
— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training StabilityN-gram memory scales capacity with negligible additional per-token FLOPs, while deterministic addressing enables host-memory offloading and asynchronous prefetching
— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
Also called Local-context N-gram embedding lookup, N-gram Embedding, N-gram embedding memory, N-gram embeddings.