Open Models Almanac
Methodsmodel architecturecontext capacity

general family · filed under model architecture

N-gram embedding lookup memory

N-gram embedding lookup memory is an embedding-based memory that scales model capacity by retrieving stored vectors with short n-grams as keys.1 It generalizes unigram lookup by conditioning retrieval on local context rather than token identity alone, and injects the retrieved embeddings into the residual stream.2

Trade-off It scales capacity with negligible additional per-token FLOPs, and its deterministic addressing enables host-memory offloading and asynchronous prefetching.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. short n-grams ending at each token serve as keys into embedding tables, and the retrieved vectors augment the corresponding token representation — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
  2. N-gram embeddings further generalize unigram lookup by conditioning memory retrieval on local context rather than token identity alone — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
  3. N-gram memory scales capacity with negligible additional per-token FLOPs, while deterministic addressing enables host-memory offloading and asynchronous prefetching — On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability

Also called Local-context N-gram embedding lookup, N-gram Embedding, N-gram embedding memory, N-gram embeddings.

sources
5
model
1
lab adopt it
1
strongest
core

Documented in

Alibaba (Qwen)

Evidence

5 quotes from 5 sources, one card per model, each opening with its strongest.

How sources treat it
core 5

Qwen3.8-Flash-Next

Alibaba (Qwen)adopted5 sources

By indexing with short n-grams, this approach makes parameter scaling highly efficient for memory-constrained accelerators without sacrificing quality.

coreunclearin Qwen3.8-Flash-NextQwen Team
4 more from this model

short n-grams ending at each token serve as keys into embedding tables, and the retrieved vectors augment the corresponding token representation.

coremodel architecturein Qwen3.8-NextQwen Team

N-gram Embedding looks up a table using the local context to scale model capacity with very little extra computation

coremodel architecturein Qwen3.8-Flash-NextQwen Team

Lookups are performed using the local context formed by "the current token plus several preceding tokens", providing additional representations for common phrases and local patterns

coreunclearin Qwen3.8-Flash-NextQwen Team

a 51B lookup memory adds capacity with little per-token compute

coremodel architecturein Qwen3.8-Flash-NextQwen Team
↑