specific method · filed under model architecture
N-gram embedding lookup
Short n-grams from the local context index embedding tables whose retrieved vectors augment token representations, adding model capacity with little extra computation.
Also called Local-context N-gram embedding lookup, N-gram Embedding, N-gram embedding lookup memory, N-gram embedding memory, N-gram embeddings, n-gram table.
- sources
- 6
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Evidence
6 spans quoted from the sources, strongest treatment first.
Lookups are performed using the local context formed by "the current token plus several preceding tokens", providing additional representations for common phrases and local patterns
Twenty million bigram and trigram embeddings are applied at layer 2, contributing 51B parameters — more than 40% of the checkpoint's non-MTP weight — while adding a table lookup rather than a matrix multiply.
short n-grams ending at each token serve as keys into embedding tables, and the retrieved vectors augment the corresponding token representation.
a 51B lookup memory adds capacity with little per-token compute
By indexing with short n-grams, this approach makes parameter scaling highly efficient for memory-constrained accelerators without sacrificing quality.
N-gram Embedding looks up a table using the local context to scale model capacity with very little extra computation
Filed alongside
Other methods under model architecture :: context capacity.