Open Models Almanac

Memory · M1

Conditional memory: lookup tables beside the experts

Tables addressed by the last few tokens add parameters for almost no arithmetic and can live outside GPU memory; they hold static patterns rather than computation, and their gains in loss carry to benchmarks unevenly

Illustration of a tall wall of hundreds of small dark-blue drawers, like an old library card catalogue. A slip of paper lies on a small teal reading stand in front of it, and three thin gold threads run from the slip to three drawers on different rows that stand slightly open. A small teal table to the left holds a single tool.

DeepSeek-V4.1-Flash is announced as a 552-billion-parameter model, but counting everything it stores, it holds about 763 billion parameters. 196.6 billion of them sit in two tables that no matrix multiplication ever reads in full. At two layers of the network, each token picks 24 rows out of a table of 384 million, using nothing but a hash of itself and the three tokens before it. The rows come to 12.7 KB per token for both layers. Turning them into something the layer can use costs about 2% of the token’s arithmetic (computed below from the configuration).

DeepSeek calls this conditional memory, and its paper on the module, Engram, presents it as “a complementary sparsity axis” to mixture-of-experts (MoE). C2 explains how an MoE layer holds many small feed-forward networks, the experts, and runs a few of them per token, so parameters grow faster than arithmetic. A lookup table goes further: a row costs no arithmetic at all until it is used. And because the row is chosen by the tokens rather than by the network’s hidden state, the system knows which rows it needs before the layer runs. GPU memory (HBM) is fast and scarce, about 140 GiB on an H200; the server’s host memory is larger but reached over a link (PCIe) with about a seventy-fifth of the bandwidth. A row whose address is known early can cross that link while the GPU works on earlier layers. Qwen3.8-Flash-Next does the same with 51 billion parameters at one layer, “held off the accelerator”. Google’s small Gemma 4 models use a simpler relative, per-layer embeddings.

Of the 24 labs whose models this site tracks, two ship a lookup memory in a tracked model, DeepSeek and Qwen (configs and code checked for 39 of the 46 models; seven robotics models have none to check), and both shipped it in 2026. Memory is close to free in arithmetic and GPU memory, but it stores fixed patterns of a few tokens, it lowers the loss more reliably than it raises every benchmark, and its parameters are parameters the experts no longer get.

Two kinds of sparse parameter

Every language model already has one lookup table. The input embedding is a matrix with one row per vocabulary entry, and the first thing a model does with a token is copy out its row. No multiplication happens: reading a row of width dd costs dd values of memory traffic and nothing else. The primer 00 counts it as a parameter that does no matrix multiplication.

An MoE layer is a different kind of sparsity. A learned router picks a few experts per token from the token’s hidden state. The parameters a token does not use are idle, but the ones it uses are multiplied in full. Following the Engram paper, write PtotP_{\mathrm{tot}} for the total trainable parameters (without the input embedding and output head), PactP_{\mathrm{act}} for those used per token, and Psparse=Ptot−PactP_{\mathrm{sparse}} = P_{\mathrm{tot}} - P_{\mathrm{act}} for the idle ones. MoE grows PsparseP_{\mathrm{sparse}} by adding experts. A lookup table grows it by adding rows, and a token touches a fixed number of rows however large the table gets.

The Engram paper’s case for rows rather than more experts rests on named entities and set phrases. A multi-token name like “Diana, Princess of Wales” is spread over several tokens, and the model has to assemble its meaning across layers; in the probing study the paper reproduces (Ghandeharioun et al. 2024), the full name can be read out of the hidden state only from layer 6. The paper calls this “an expensive runtime reconstruction of a static lookup table”. If the meaning of a frequent short sequence were simply stored, the early layers would be free for other work.

Statistical language models did exactly that for decades. An n-gram is a run of nn consecutive tokens, and an n-gram model stores counts for every n-gram it has seen, backing off to shorter ones for those it hasn’t (Katz 1987; Kneser and Ney 1995). The modern versions store a learned vector instead of a count and leave the backing off to a learned gate. What they lose is the count: a backoff model knows when an n-gram is unseen, while a hashed table hands an unseen n-gram the rows other n-grams trained.

Addressing by content or by tokens

There are two ways to choose which rows a token reads, and the choice decides where the table can live.

A content-addressed memory computes a query from the hidden state and looks for the nearest keys. Lample et al.’s product-key memory (2019) is the standard example. Every key is a pair of sub-keys from two sets of Nkeys\sqrt{N_{\text{keys}}}, so the best few sub-keys in each half determine the best full keys: about 2Nkeys2\sqrt{N_{\text{keys}}} comparisons instead of NkeysN_{\text{keys}}. A 12-layer model with one such memory beat a 24-layer model without one, and Meta later scaled the idea to 128 billion memory parameters (Berges et al. 2024). Because the query comes from the hidden state, the address is unknown until the layer before the memory has finished. Both papers found the memory works best in the middle of the network (Lample et al.: worst at layer 1 of 6, best at 4 or 5), because a good query needs context.

A token-addressed memory computes the row index from the token ids alone. The address of every row a sequence will read is fixed the moment the tokens exist, which is what lets Engram and Qwen fetch rows from host memory while the GPU is busy. The price is that the address cannot depend on meaning: the n-gram “by the bank” reads the same rows whether the paragraph is about a river or a loan. Only the hidden state, which has read the paragraph, can tell them apart, and that is the job of Engram’s gate.

Hashing n-grams into a table

Engram, Qwen’s layer and OverEncoding (Huang et al. 2025) key their rows on the n-gram that ends at the current token: for order nn, the tokens at positions t−n+1t-n+1 to tt. Engram calls it the suffix n-gram. It includes the current token, so it is causal and it is known as soon as that token is. (N-Grammer, Roy et al. 2022, keyed bigrams of learned cluster ids instead, with token bigrams as a variant.)

The tables can’t have one row per n-gram. DeepSeek-V4.1-Flash’s tokenizer has 129,280 entries, so there are 1.7×10101.7 \times 10^{10} possible bigrams and 2.2×10152.2 \times 10^{15} trigrams (computed). The answer is the hashing trick of linear models (Vowpal Wabbit, Langford et al. 2007; hash kernels, Shi et al. 2009; analysed by Weinberger et al. 2009): hash each n-gram to one of MM rows and accept that some n-grams share a row. fastText hashed word bigrams the same way (Joulin et al. 2016). Training then makes each shared row the average of what its n-grams would want, weighted by how often each occurs. In a table of 16 million rows, the other n-grams in a given row account on average for about one token in 16 million, so an n-gram more frequent than that owns its row, and a rarer one reads mostly its neighbours’ average (our derivation, for a quadratic loss near the optimum).

Three refinements reduce the damage.

The first is tokenizer compression. Subword tokenizers give “Apple”, “apple” and ” apple” different ids. Engram maps each token id to a canonical id first (the paper names NFKC, a Unicode rule that merges look-alike characters, and lowercasing; the vLLM code also strips accents and collapses whitespace) and hashes the canonical ids. For V4.1 the config gives the result, 99,092 ids out of 129,280 (engram_compressed_vocab_size: 99092), the 23% reduction the paper reports for its 128k tokenizer. Compression is a collision by design: it merges n-grams that differ only in case or accents, so they share what they learn, and it shrinks the space of possible 4-grams to about a third (computed). In the paper’s ablation, removing it is among the three changes that hurt most. Qwen tried the same normalization for Qwen3.8-Flash-Next and “observed no consistent performance gains”, and its layer hashes raw ids.

The second is several hash heads. Each order gets KK heads, each hashing the n-gram into its own table, and the retrieved rows are concatenated. Hash embeddings (Svenstrup et al. 2017), which Engram cites for this, give the argument. With one hash into MM rows, each of the other N−1N-1 distinct n-grams misses a given n-gram’s row with probability 1−1/M1-1/M, so the row is shared with probability 1−(1−1/M)N−1≈1−e−N/M1-(1-1/M)^{N-1} \approx 1 - e^{-N/M}. Several heads don’t make a row less crowded. They give each n-gram a combination of rows that almost no other n-gram shares: with independent hashes, a given pair collides in all KK heads with probability (1/M)K(1/M)^K. V4.1 uses three orders (2, 3 and 4) with eight heads each, so a token reads 24 rows per Engram layer from 24 head tables of just over 16 million rows, 384 million in all (engram_vocab_size: 16000000, engram_n_heads: 8, engram_max_ngram_size: 4).

The third is a prime table size. OverEncoding found that when the table size was an exact multiple of the vocabulary size, most of its gain disappeared. If a bigram’s id is aV+baV + b for vocabulary size VV and the table has kVkV rows, the row depends only on bb and on a mod ka \bmod k, so most of the first token is ignored. A prime size avoids such alignments. In V4.1 the config’s row counts are exactly the sums of 24 consecutive primes above 16,000,000 per layer (engram_num_embeddings: [384006168, 384016682]), which the vLLM code draws in order and never reuses.

The shipped heads are built differently from independent hashes but behave like them. The vLLM code for V4.1 and the Transformers code for Qwen multiply each token id by a fixed odd number, combine the products with exclusive-or into one non-negative 63-bit value per n-gram, and reduce that value modulo eight different primes. Two n-grams with values x≠yx \neq y share head kk‘s row exactly when the prime pkp_k divides x−yx - y. Any two primes multiply to far less than 2632^{63}, so two heads act like independent hashes. Any three multiply to more than 2632^{63}, while ∣x−y∣<263|x - y| < 2^{63}, so three cannot all divide the difference: two different values share at most two of their eight rows. All eight are shared only if the 63-bit values are equal, about 0.05 pairs among 10910^9 n-grams if the values behave like random numbers (computed). The construction is the “Chinese remainder partition” of compositional embeddings (Shi et al. 2020).

A toy: a 48-row table, heads x mod 47, 43, 41

heads
head 1: x mod 478 of 20 n-grams share their row herehead 2: x mod 439 of 20 n-grams share their row herehead 3: x mod 418 of 20 n-grams share their row here48 rows; head k reaches rows 0 to p − 1 (emptycells at the right: out of its reach). Coraloutline: a row holding two or more n-grams;dark dot: an n-gram that shares its row inevery head used with one same other n-gram.Ids below 41 × 43 × 47 = 82,861.head 1: x mod 478 of 20 n-grams share their row herehead 2: x mod 439 of 20 n-grams share their row herehead 3: x mod 418 of 20 n-grams share their row here48 rows; head k reaches rows 0 to p − 1 (empty cells at the right: out of its reach). Coral outline:a row holding two or more n-grams; dark dot: an n-gram that shares its row in every head used withone same other n-gram. Ids below 41 × 43 × 47 = 82,861.

n-grams sharing all their rows with another: 0; sharing two rows: 0 (of 20)

The real table

one head of the V4.1 layer-1 table, M =16,000,057 rows00.250.50.7511051061071081091010distinct n-grams N (log axis)probabilityN = Mshares its row in one headone head of the V4.1 layer-1 table, M = 16,000,057 rows00.250.50.7511051061071081091010distinct n-grams N (log axis)probabilityN = Mshares its row in one head

N = 108 distinct n-grams: a given n-gram shares its row in one head with probability 0.9981
chance a given n-gram shares two of its eight rows with some other n-gram: 1.094 × 10−5
expected pairs of n-grams sharing two rows: 547
sharing three or more of eight, in the shipped hash: impossible (any three of the primes multiply to more than 263)

Toy ids are random integers below 41 × 43 × 47 = 82,861, so, as in the shipped code, two different ids can never share all three rows. The real curve is 1 − exp(−(N − 1)/M) for one head. The shipped hash reduces one 63-bit value per order by eight different primes, and any three of them multiply to more than 263, so two different values share at most two of their eight rows.

Figure 1.Hashing many n-grams into a small table. With one head, a row is soon shared; with several heads hashed by different primes, two n-grams almost never share all their rows. In the shipped code for DeepSeek-V4.1-Flash the heads reduce one 63-bit hash by eight primes, so two different hash values can share at most two of their eight rows. In the toy, ids are kept below 41 × 43 × 47 so the same holds.

The gate: letting the context veto the lookup

A hashed row is a context-free guess. It is shared with other n-grams, and the same n-gram can mean different things in different places. Engram therefore lets the hidden state decide how much of it to use. Write ete_t for the concatenated rows at position tt and hth_t for the hidden state there (the Engram paper’s notation; elsewhere in this series hh counts heads), which by then has seen the whole preceding context through attention. The module computes a key and a value from the memory and a scalar gate from the match between key and hidden state (Engram, Eqs. 3 and 4):

kt=WKet,vt=WVet,αt=σ ⁣(RMSNorm(ht)⊤RMSNorm(kt)d),k_t = W_K e_t,\qquad v_t = W_V e_t,\qquad \alpha_t = \sigma\!\left(\frac{\mathrm{RMSNorm}(h_t)^\top \mathrm{RMSNorm}(k_t)}{\sqrt d}\right),

and adds αtvt\alpha_t v_t to the residual stream. Here dd is the model width (5,120 in V4.1), and RMSNorm divides a vector by its root-mean-square and rescales it. The sigmoid is softmax attention over two slots, the memory with score ss and an empty slot with score zero: σ(s)=es/(es+e0)\sigma(s) = e^s/(e^s + e^0). A softmax over the memory alone would always be 1. In the paper’s case study the gate opens at the end of names and set phrases (“Alexander the Great”, “By the way”), for the residual stream the authors found easiest to read.

Nothing in the lookup itself is trained. The hash and the normalization are fixed functions, so no gradient decides which rows a token reads, and unlike an MoE router there is nothing to balance. What learns is the rows, the two projections and, through them, the gate. Its gradient has a useful form: with u=αvu = \alpha v, the gradient reaching the rows is scaled by α\alpha through the value and by α(1−α)\alpha(1-\alpha) through the key, so an occurrence with the gate nearly shut sends almost no gradient into the rows it read. A row shared by two n-grams is shaped mostly by the contexts in which the gate opens, and the key learns to open it where the value helps (our derivation). Apart from that one scalar, what the module adds at a position is a function of the last four token ids alone. In the paper’s ablations, removing the gate is among the three changes that hurt most, with tokenizer compression and the per-stream keys described next.

The paper’s models, like both shipped ones, widen the residual stream into four parallel copies (R2 explains these hyper-connections). The module then computes one key per copy and one shared value, so each copy gets its own gate: wkv in V4.1 maps the 6,144 retrieved values to 5×5,1205 \times 5{,}120 outputs, four keys and a value. One change appears in the shipped code: the vLLM code for V4.1 and the Transformers code for Qwen both pass the score through a signed square root before the sigmoid, σ(sign(s)∣s∣)\sigma(\mathrm{sign}(s)\sqrt{|s|}), which compresses large scores; the vLLM comment says this matches DeepSeek’s training kernel.

The Engram paper also runs the gated values through a short causal convolution (four taps, spaced by the largest n-gram order) to widen the window a little. Removing it “only marginally degrades performance” in the paper’s ablation, and DeepSeek dropped it for V4.1 because “its performance gains do not justify the added complexity in our inference stack”. Qwen kept it (ple_conv_kernel_size: 4; the Transformers code initialises it to zero). Either way a request carries the previous token ids for the hash through decoding (three for V4.1, two for Qwen, plus a short convolution history); vLLM keeps V4.1’s beside its attention cache, so chunked prompts, cached prefixes and rejected draft tokens see the right history.

The arithmetic sits in the projection. In V4.1, wkv has 157.3 million parameters per Engram layer, 315 million for the two. That is about 2% of an output token’s arithmetic (16 billion active, from the config) and about 4% of a prompt token’s (8 billion), the cost that matters when many tokens are processed at once. When few are, the cost is reading those 315 MB of FP8 weights on every step, about 3% of what a one-token step reads (computed). In Qwen3.8-Flash-Next the projections are 32.8 million parameters, about 0.5% of the active count.

Figure 2.One Engram layer in DeepSeek-V4.1-Flash. The rows are chosen from the last four tokens alone, so they can be fetched before the layer runs; only the gate needs the hidden state. Each of the four residual copies gets its own key and gate and shares one value.

Where in the network

Three needs pull the placement in different directions. The lookup is meant to replace the early layers’ work of assembling short patterns, which argues for putting it early. The gate needs a hidden state that has seen some context, which argues against layer 0. And the rows have to arrive from host memory while earlier layers run, which argues for later.

The Engram paper’s ablation on a 12-layer, 3-billion-parameter MoE trained for 100 billion tokens found layer 2 best for a single module with a 1.6-billion-parameter table (validation loss 1.770 against 1.808 without memory), with performance “degrading as the insertion point moves deeper”. The authors’ reading: “one round of attention is already sufficient to provide a meaningfully contextualized hth_t for gating, while still being early enough to replace the backbone’s bottom-layer local aggregation”. Splitting the same table over layers 2 and 6 was slightly better again (1.768). N-Grammer, which adds its bigram embeddings without a gate, found the input layer best and every later position worse. Together with the product-key results, this suggests that token-addressed memories work best early and content-addressed ones in the middle, as their addressing would predict.

Qwen’s ablation for Qwen3.8-Flash-Next, under a fixed table budget, came out flatter: “No single depth regime consistently dominates.” Loss stayed between 1.541 and 1.544 for every single-layer placement from layer 1 to layer 25, and the nine-benchmark average between 46.6 and 47.9, against 1.585 and 45.4 without the table. Two layers instead of one brought “no consistent benefit”. Qwen put its single layer at layer 2, “allowing host-memory prefetching to overlap with the computation of the first layer”.

DeepSeek-V4.1-Flash has two modules, at layers 1 and 14 of 40 counting from zero (engram_layer_ids: [1, 14]), apparently the same pattern as the paper’s 27B model at layers 2 and 15 of 30, which by the paper’s wording count from one. The report’s stated reason is neither of the above: they are placed “to balance memory usage across training pipeline stages”. A model this size is split across many GPUs. Pipeline parallelism gives each group of GPUs a consecutive block of layers, a stage; tensor parallelism splits each layer’s matrices so that several GPUs compute one layer together. A table counts against its stage’s memory. Layer 14 is not early; it is the 15th of 40.

How much memory, how many experts

If memory and experts both add idle parameters, a fixed budget of them must be split. The Engram paper defines ρ\rho as the share of PsparseP_{\mathrm{sparse}} that goes to experts, so ρ=1\rho = 1 is a pure MoE, and sweeps it at two small compute budgets with about ten times as many total as active parameters (5.7 and 9.9 billion total). Validation loss forms a U: pure MoE is not best, and moving 20 to 25% of the sparse budget into the table gives the lowest loss. At the larger budget, loss goes from 1.7248 to 1.7109, a difference of 0.0139. Even at ρ≈40%\rho \approx 40\%, with fewer than half the experts, the Engram model matches the pure MoE. At the other end of the U, “memory cannot replace computation”.

The paper then trains a 27B model for 262 billion tokens with 55 instead of 72 routed experts and a 5.7-billion-parameter memory (ρ=74.3%\rho = 74.3\%), against an MoE of the same total size. Validation loss drops by 0.012. The benchmarks rise by 3.0 points on MMLU, 5.0 on BBH, 3.7 on ARC-Challenge and 3.0 on HumanEval; no repeated runs are reported. (The abstract says MMLU +3.4; its own table gives +3.0.) The closed-book factual benchmarks gain less than reasoning: TriviaQA by 1.9, PopQA by 0.2. By our reading of the paper’s configuration table, its heads have about 2.3 million rows each, a seventh of V4.1’s, so these gains came from heavily shared rows.

Qwen ran the same experiment with a different result. In its fixed-budget ablation (Table 8 of the Qwen3.8-Flash-Next report), trading experts for table rows lowered the loss most at 25% of the budget, “consistent with the allocation sweet spots reported in prior work”, but “downstream benchmarks show no clear improvement over the MoE-only baseline”. Qwen concluded that the two “play distinct roles” and added the table on top of a fixed MoE instead.

Adding on top is the experiment behind a sentence in the report’s abstract. Qwen grew the table from 20 to 200 times what the report calls its 250,000-entry vocabulary (248,320 in the config) with the experts held fixed (Table 9); the report doesn’t say how these multiples relate to the shipped model’s 20 million rows per head. In the abstract’s words, “enlarging the n-gram vocabulary lowers loss monotonically while downstream accuracy saturates”. The table itself is more two-sided. Every size beats no table on all ten benchmarks. The two Chinese knowledge benchmarks keep rising (C-Eval from 71.75 at 20× to 74.94 at 200×), MATH and GSM8K peak at the smallest size and fall back, and MMLU barely moves after 50×. The ablation model’s size isn’t stated.

OverEncoding gives the third data point, with hashed n-gram embeddings summed into the input embedding of an MoE (OLMoE): the loss fell by about the same amount at 1.3 and 7 billion parameters (0.082 and 0.076), while the downstream average gained 0.014 and then only 0.007. The authors suggest the experts “may overlap with the benefits provided by sparse embedding parameters”.

Training lossloss (zoomed axis, not from zero)1.501.521.541.561.581.60None20×50×100×200×Loss at None: 1.585Loss at 20×: 1.553Loss at 50×: 1.541Loss at 100×: 1.534Loss at 200×: 1.5261.5261.585Chinese knowledgechange from no table, points−20+2+4+6+8None20×50×100×200×C-Eval at None: 66.91 (0 from None)C-Eval at 20×: 71.75 (+4.84 from None)C-Eval at 50×: 72.12 (+5.21 from None)C-Eval at 100×: 73.75 (+6.84 from None)C-Eval at 200×: 74.94 (+8.03 from None)C-Eval+8.03CMMLU at None: 68.10 (0 from None)CMMLU at 20×: 72.29 (+4.19 from None)CMMLU at 50×: 72.48 (+4.38 from None)CMMLU at 100×: 72.73 (+4.63 from None)CMMLU at 200×: 73.24 (+5.14 from None)CMMLU+5.14Mathschange from no table, points−20+2+4+6+8None20×50×100×200×MATH at None: 32.52 (0 from None)MATH at 20×: 37.38 (+4.86 from None)MATH at 50×: 37.32 (+4.80 from None)MATH at 100×: 36.98 (+4.46 from None)MATH at 200×: 35.34 (+2.82 from None)MATH+2.82GSM8K at None: 59.21 (0 from None)GSM8K at 20×: 65.09 (+5.88 from None)GSM8K at 50×: 64.00 (+4.79 from None)GSM8K at 100×: 63.08 (+3.87 from None)GSM8K at 200×: 62.96 (+3.75 from None)GSM8K+3.75English knowledgechange from no table, points−20+2+4+6+8None20×50×100×200×MMLU at None: 62.78 (0 from None)MMLU at 20×: 64.14 (+1.36 from None)MMLU at 50×: 64.71 (+1.93 from None)MMLU at 100×: 64.70 (+1.92 from None)MMLU at 200×: 64.85 (+2.07 from None)MMLU+2.07MMLU-Pro at None: 33.43 (0 from None)MMLU-Pro at 20×: 34.46 (+1.03 from None)MMLU-Pro at 50×: 35.80 (+2.37 from None)MMLU-Pro at 100×: 35.87 (+2.44 from None)MMLU-Pro at 200×: 35.21 (+1.78 from None)MMLU-Pro+1.78x axis: n-gram table size as a multiple of the250K base vocabulary (categories, evenlyspaced, not to scale)Training lossloss (zoomed axis, not from zero)1.501.521.541.561.581.60None20×50×100×200×Loss at None: 1.585Loss at 20×: 1.553Loss at 50×: 1.541Loss at 100×: 1.534Loss at 200×: 1.5261.5261.585Chinese knowledgechange from no table, points−20+2+4+6+8None20×50×100×200×C-Eval at None: 66.91 (0 from None)C-Eval at 20×: 71.75 (+4.84 from None)C-Eval at 50×: 72.12 (+5.21 from None)C-Eval at 100×: 73.75 (+6.84 from None)C-Eval at 200×: 74.94 (+8.03 from None)C-Eval+8.03CMMLU at None: 68.10 (0 from None)CMMLU at 20×: 72.29 (+4.19 from None)CMMLU at 50×: 72.48 (+4.38 from None)CMMLU at 100×: 72.73 (+4.63 from None)CMMLU at 200×: 73.24 (+5.14 from None)CMMLU+5.14Mathschange from no table, points−20+2+4+6+8None20×50×100×200×MATH at None: 32.52 (0 from None)MATH at 20×: 37.38 (+4.86 from None)MATH at 50×: 37.32 (+4.80 from None)MATH at 100×: 36.98 (+4.46 from None)MATH at 200×: 35.34 (+2.82 from None)MATH+2.82GSM8K at None: 59.21 (0 from None)GSM8K at 20×: 65.09 (+5.88 from None)GSM8K at 50×: 64.00 (+4.79 from None)GSM8K at 100×: 63.08 (+3.87 from None)GSM8K at 200×: 62.96 (+3.75 from None)GSM8K+3.75English knowledgechange from no table, points−20+2+4+6+8None20×50×100×200×MMLU at None: 62.78 (0 from None)MMLU at 20×: 64.14 (+1.36 from None)MMLU at 50×: 64.71 (+1.93 from None)MMLU at 100×: 64.70 (+1.92 from None)MMLU at 200×: 64.85 (+2.07 from None)MMLU+2.07MMLU-Pro at None: 33.43 (0 from None)MMLU-Pro at 20×: 34.46 (+1.03 from None)MMLU-Pro at 50×: 35.80 (+2.37 from None)MMLU-Pro at 100×: 35.87 (+2.44 from None)MMLU-Pro at 200×: 35.21 (+1.78 from None)MMLU-Pro+1.78x axis: n-gram table size as a multiple of the 250K base vocabulary (categories, evenly spaced, notto scale)
Figure 3.Growing the n-gram table on top of a fixed set of experts, in Qwen's ablation for Qwen3.8-Flash-Next. Loss falls at every size. Every size beats no table on every benchmark, but beyond the smallest table the Chinese knowledge benchmarks keep rising, maths falls back and MMLU stays flat. No repeated runs are reported, and the report does not give the ablation model's size.

The Engram paper also tested what the table holds, by setting the memory’s output to zero at inference without retraining. Factual benchmarks kept only 29 to 44% of their scores (TriviaQA 29%), reading comprehension 81 to 93%. This suggests the model keeps much of its factual knowledge in the table, though its factual benchmarks gained less than reasoning from having one.

Where the table lives

Serving a model runs in two phases. Prefill processes a whole prompt at once: thousands of tokens share each read of the weights, so the GPU is limited by arithmetic. Decode then produces one token per step for each request in a batch, and each step must read from GPU memory the weights of every expert the batch touches. That read, not the arithmetic, sets the step time; the step is memory-bound. A larger batch touches more distinct experts and so reads more weights. Each request also keeps the keys and values of its earlier tokens in GPU memory, the KV cache, so the memory left after the weights decides how many requests fit.

V4.1’s two tables, with their FP8 rows (one byte per value) and 8 bytes of scales per 256-value row, take 188.8 GiB (computed). One H200 has 140.4 GiB of memory (NVIDIA lists 141 GB; we take 143,771 MiB as reported by the driver), so the tables alone are 1.34 H200s. Qwen’s table is 95.4 GiB in BF16, two bytes per value, the format SGLang measured.

Each token reads very little of it: 12.7 KB for V4.1 across both layers, 5 KB for Qwen. The worked example below is vLLM’s recipe for V4.1 on four H200s with four-way tensor parallelism. There each GPU holds a quarter of the hash heads and reads only those rows, over its own PCIe Gen5 x16 link at a nominal 64 GB/s (all hardware figures here are vendor nominals), from pinned host memory, which is host memory locked in place so the GPU can read it directly. Take 8 sequences with speculative decoding and the recipe’s 5-token drafts (P1): a step then carries up to 48 tokens. The drafter has no Engram module, but every draft token checked by the full model needs its own rows.

The timing is tighter than “known ahead of time” suggests. The n-gram includes the current token, so a decode step’s rows are known only once the previous step has produced its token. When the tables are offloaded, the vLLM code computes all hashes at the start of the forward pass and starts the lookup for both Engram layers on a side stream, a second queue of GPU work that runs beside the main one. The rows for layer 1 have the embedding and layer 0 to arrive in; layer 14’s rows have fourteen layers; Qwen’s layer has one. For 48 tokens, each GPU’s share of layer 1’s rows is 76 KB, about 1 µs at the nominal bandwidth, while layer 0 takes about 195 µs to read its weights (computed, assuming routing spreads tokens evenly over experts).

As the number of tokens in a step grows, the rows grow with it, while layer 0 stops growing once every expert is touched. From about 4,000 tokens the layer is limited by arithmetic, and from there on rows and arithmetic both grow in proportion to the tokens: the rows settle at about a quarter of layer 0’s time, however long the prompt (computed at nominal dense FP8 throughput; V4.1’s prompt tokens run only through the first 20 of its 40 layers, which is why the report counts 8 billion active parameters per prompt token and 16 billion per output token). vLLM’s TP4 command for Blackwell GPUs caps a step at 4,096 tokens.

link
0100200layer 0 on one GPU (memory-bound)195.2 µslayer-1 Engram rows for that GPU overPCIe, own link, lower bound at nominallink1.19 µstime per GPU, µs (linear, from zero; nominal,computed)0100200layer 0 on one GPU (memory-bound)195.2 µslayer-1 Engram rows for that GPU over PCIe, own link, lower bound at nominal link1.19 µstime per GPU, µs (linear, from zero; nominal, computed)

48 tokens in this step (an example decode step: 8 sequences × 6 tokens, each 1 token plus a 5-token speculative draft): layer 0 takes 195.2 µs per GPU, memory-bound (weight reads 195.2 µs, arithmetic 4.54 µs); the rows are 76.0 KB per GPU, at least 1.19 µs over PCIe, own link. Rows ÷ layer 0 = 0.61%.

rows ÷ layer 0 (linear, from zero) againsttokens n (log axis)0%10%20%30%40%1485124,09616,384rows ÷ layer 0 (linear, from zero) against tokens n (log axis)0%10%20%30%40%1485124,09616,384

DeepSeek-V4.1-Flash on 4 H200s, tensor-parallel degree fixed at 4: each GPU fetches its own heads' rows. Layer 0 takes the longer of its weight reads (4.8 TB/s HBM) and its arithmetic (peak dense FP8,1,979 TFLOPS per GPU). Uniform routing over the experts; nominal link bandwidth, so the row time is a lower bound. Our computation, not a measurement.

Figure 4.Can the rows arrive in time? For DeepSeek-V4.1-Flash on four H200s, each GPU fetches its share of the first Engram layer's rows while layer 0 runs. At 48 tokens (8 sequences with 5-token drafts) the rows take under 1% of layer 0's time. Their share grows with the tokens in a step (14% at 2,048) until, from about 4,000 tokens, the step is bound by arithmetic and the share stays at about a quarter, however many tokens follow. Nominal bandwidths and peak throughput, uniform routing, each GPU with its own host link: our computation, a lower bound on the row time, not a measurement.

Those times are lower bounds. The engines read rows with a GPU kernel that loads host memory directly (zero-copy), so each lookup pays a PCIe round trip, occupies GPU compute units while it waits, and moves 264-byte rows below the link’s peak. vLLM’s two Engram layers take turns “instead of jointly starving decoder compute of SMs”, and on Blackwell GPUs a step of 4,096 tokens or more runs its lookup inline, because “big lookups stall persistent main-stream kernels anyway”. Where two GPUs share a PCIe uplink, the row times double, and one CPU socket feeding all four can slow them further. And sharding by head adds a cost the bus doesn’t show: in vLLM’s tensor-parallel path, the rows and the projection’s output are gathered across the GPUs over NVLink before the gate, about 200 MB per GPU per Engram layer for a 4,096-token chunk, more than the host read (our estimate from the vLLM code).

The one engine measurement finds no throughput cost at low load. SGLang keeps each GPU’s shard of Qwen’s table in pinned host memory and gathers the 16 rows per token with a zero-copy kernel. On an H200 with four-way tensor parallelism, offload cut the weights per GPU from 83.91 to 60.45 GiB and raised the KV-cache capacity from 1.84 to 3.28 million tokens (+78.54%), while throughput at 1, 2 and 4 concurrent requests changed by −0.07% as a geometric mean (SGLang’s measurement, BF16, with speculative decoding, at loads where decode is bound by weight reads). The gain is cache space, which becomes throughput only when cache runs short. The Engram paper measured a 100-billion-parameter table in host memory, prefetched over PCIe behind the first block: throughput fell 1.9% and 2.8% on dense 4B and 8B models on an H800. The table was inserted for the measurement rather than trained, as far as the paper says, the backbones were dense, and the number a throughput average, not a latency.

The engineering limits appear in recipes and code rather than in the reports. vLLM’s Qwen recipe says the n-gram layer “does not initially support pipeline parallelism”, and offload “currently only runs on Nvidia devices”. It requires offload on H100s, whose 80 GB leave no room for the table, and when attention is data-parallel and experts are spread across GPUs. For that mode the V4.1 code can also split the tables by row over a node’s GPUs, each reading its tokens’ rows from its peers’ memory, or keep one shared copy in host memory per node. Otherwise V4.1 needs about 189 GiB of pinned host memory per serving instance, and Qwen “at least 51 GB plus runtime headroom” in its FP8 build. During reinforcement-learning rollouts DeepSeek keeps the tables “resident in GPU memory”.

Training a table whose rows are rarely touched

A table row receives a gradient only when one of its n-grams occurs in the batch. SCONE (Yu et al. 2025) put a number on how sparse that gets for plain token embeddings: after 100 million training tokens, 97.6% of the entries of a 32K vocabulary had received more than 100 updates, against 7.3% of a 2M vocabulary. A hashed n-gram table with tens of millions of rows per head is far sparser. Older memory papers treated the tables differently for this reason: Lample et al. gave the product-key values a four times higher Adam learning rate “since the memory values are learned with sparse updates”, and N-Grammer used Adagrad, whose per-parameter step size lets rarely seen features take large steps.

The labs differ:

  • The Engram paper trains its tables with Adam at five times the base learning rate and no weight decay, while the backbone uses Muon, an optimizer that orthogonalizes the update of each weight matrix. It gives no reason.
  • Qwen runs the table on “Adam with weight decay disabled”, and the key and value projections on Muon.
  • DeepSeek dropped Adam for V4.1 because it “substantially increases the optimizer-state memory footprint”. Adam keeps two numbers per parameter, 1.57 TB in FP32 for 196.6 billion parameters; a momentum buffer alone is half that (computed). V4.1 uses a momentum update followed by Sinkhorn balancing, which rescales the update so that its rows and columns have balanced norms, with no weight decay, the learning rate scaled by five as in the paper, and near-zero rows masked. The same update trains the token embedding and the output head.

All three train the table without weight decay, and there is a reason. Weight decay shrinks every row at every step, while a rare row is pushed back only when it occurs. With AdamW-style decay (each step shrinks a row by learning rate × λ\lambda), a row seen once every KK steps settles at about c/(λK)c/(\lambda K) per coordinate, cc the normalized update, whatever the learning rate (our derivation). A higher learning rate can’t offset decay; it lets a row seen a few hundred times in the whole run travel far enough. A dense momentum update also keeps moving a row on stale momentum after its last gradient; masking near-zero rows, as V4.1 does, keeps such leftovers from being normalized into full-size steps (our reading).

In training, the table is partitioned by row across dedicated groups of GPUs, rows for the whole local batch are fetched before each pipeline stage starts, and gradients are buffered and sent back to the owning GPUs after the backward pass (V4.1 report).

Per-layer embeddings: the small-model relative

Google’s Gemma 4 E2B and E4B (not tracked here; the tracked Gemma 4 31B has hidden_size_per_layer_input: 0) carry the simplest member of the family. The Gemma 4 report says they “use per-layer embeddings as in Gemma 3n (Gemma Team, 2025b), making them 2.3B and 4.5B effective out of 5B and 8B total parameters”. The model card: PLE “gives each decoder layer its own small embedding for every token”.

E2B has 35 layers and a per-layer width of 256 (hidden_size_per_layer_input: 256, vocab_size_per_layer_input: 262144), so its table is 262,144×35×256=2.35262{,}144 \times 35 \times 256 = 2.35 billion parameters, about half the model. The text model minus the table gives 2.28 billion, the “effective” count to within rounding (computed; the reported 5 billion total also includes the vision and audio encoders). Each token reads one row of 35×25635 \times 256 values, 17.9 KB in BF16.

Google’s documents don’t describe the mechanism; the Transformers code does. A token’s row is split into one 256-wide vector per layer, pℓ(xt)p_\ell(x_t) for layer ℓ\ell, and added to a projection of the ordinary token embedding down to 256. At the end of each layer, after attention and the feed-forward network, the hidden state is projected down to 256, passed through GELU, multiplied element by element with that layer’s vector, projected back and added to the residual stream:

h←h+RMSNorm(W↑ [ GELU(W↓h)⊙pℓ(xt) ]).h \leftarrow h + \mathrm{RMSNorm}\big(W_{\uparrow}\,[\,\mathrm{GELU}(W_{\downarrow} h) \odot p_\ell(x_t)\,]\big).

Compare a gated feed-forward block (GeGLU), W2[GELU(W1h)⊙W3h]W_2[\mathrm{GELU}(W_1 h) \odot W_3 h]: PLE is a 256-wide GeGLU whose second branch, normally a projection of the hidden state, is replaced by a vector looked up by the token. That is Engram’s move, computation replaced by a table read, in the small. It is context-dependent but not a gate in Engram’s sense, since GELU’s output is not bounded to (0, 1); it works at every layer, and it is keyed by the token alone, with no n-grams and no hashing. With only 262,144 rows there are no collisions to manage. Google’s Gemma 3n documentation says the PLE data “can be generated separately, outside the operating memory of the model, cached to fast storage”; Gemma 4’s documents only quote memory “without Per-Layer Embeddings”. On a phone, as for V4.1 on a server, a large share of the parameters stays out of the scarce memory.

The name PLE also turns up for a different design: SGLang, vLLM and Qwen’s own code call Qwen’s single hashed n-gram layer “PLE” (ple_layer_ids, VLLM_PLE_CPU_OFFLOAD).

Side by side

Engram (paper, 27B)DeepSeek-V4.1-FlashQwen3.8-Flash-NextGemma 4 E2B PLEOverEncodingProduct-key memory
Addresshashed suffix n-gramshashed suffix n-gramshashed suffix n-gramstoken idhashed n-gramshidden state (nearest keys)
Known before the layer runsyesyesyesyesyesno
Orders2, 32, 3, 42, 311 to 3none
Layers2 and 15 of 301 and 14 of 40 (from 0)layer 2 of 48 (from 1)all 35input onlyreplaces 1 or 2 FFNs
Table parameters5.7B196.6B51.2B2.35Bup to 26Bup to 1B (Lample)
Bytes read per tokennot computed12.7 KB (FP8)5 KB (BF16)17.9 KB (BF16)not computednot computed
How the context weighs itscalar gate per streamscalar gate per streamscalar gate per stream256-wide productnonesoftmax over top keys
Where the table sitsnot statedhost memory or GPUhost memory or GPUoff-accelerator (Gemma 3n docs)CPU at inferenceGPU
Table optimizerAdam, 5× LR, no decaymomentum + Sinkhorn, 5× LR, no decayAdam, no decaynot statednot statedAdam, 4× LR (values)
Reported effectloss −0.012, MMLU +3.0 (vs MoE of equal size)no ablation publishedloss down, benchmarks flat at fixed budgetnot publishedloss −0.08, downstream gain halves at 7B MoE12 layers beat 24

The first four columns are one family, differing in size (2 to 197 billion table parameters), key and placement, and all four let the hidden state weigh the lookup. Gated or not, Qwen’s layer and OverEncoding both found smaller downstream gains on an MoE than the loss suggests, so the gate is not the whole story.

Where the models in the map sit

Model (lab)Lookup memoryConfig fieldsTable parametersShare of stored parameters
DeepSeek-V4.1-Flash (DeepSeek)Engram, 2 layersengram_layer_ids: [1, 14], engram_n_heads: 8, engram_max_ngram_size: 4, engram_head_dim: 256196.6B (computed from engram_num_embeddings)26% (computed, of about 763B with drafter)
Qwen3.8-Flash-Next (Qwen)hashed n-gram layer, 1 layerple_layer_ids: [2], ngram_size: 3, heads_per_ngram: 8, ngram_vocab_size_base: 2000000051.2B (computed)28% (51 of 180B with the 4B MTP head, card)
Gemma 4 31B (Google)nonehidden_size_per_layer_input: 000
every other tracked modelnone in config or code

V4.1-Flash-1017 has the same config and code as V4.1-Flash. DeepSeek-V4-Flash and V4-Pro have no Engram; the V4 report lists “more sparse embedding modules” as future work. DeepSeek-V4’s hash routing, which sends tokens to experts in its first three MoE layers by a hash of the token id, is routing, not a table. Neither lab has published a comparison with the other’s design.

Easy to get wrong

  • DeepSeek’s 552 billion is the backbone. The 196 billion Engram parameters and a 14-billion-parameter drafter come on top, and because the experts are stored in 4 bits and the table rows in 8, the tables take about 40% of the checkpoint bytes (computed).
  • Known ahead of time means known at the start of the forward pass. The n-gram ends at the current token, so a decode step can’t fetch its rows before that token exists. The overlap comes from placing the lookup after a layer or more.
  • Offloaded is not free. The bytes are tiny, but a zero-copy read pays a PCIe round trip and holds GPU compute units while it waits, and in tensor-parallel serving the rows and the projection’s output are gathered over NVLink afterwards.
  • Lower loss is not the same gain on every benchmark. Qwen’s tables lowered the loss at every size while maths fell back, and Engram’s 27B gained several points for a loss drop of 0.012. Each result belongs to its own design, model and budget.
  • PLE names two different things: Gemma’s per-layer embeddings, a token-keyed table at every layer, and, in Qwen’s code and the serving engines, the single hashed n-gram layer.
  • DeepSeek replaced Adam for V4.1, which halves the tables’ optimizer state; the momentum buffer still holds one number per parameter.

Check yourself

Qwen3.8-Flash-Next reads 16 rows of 160 BF16 values per token. At a decode step of 64 sequences, how many bytes does its n-gram layer read, and how long does that take over a 64 GB/s link?

16 × 160 × 2 bytes = 5,120 bytes per token; × 64 = 327,680 bytes, about 0.33 MB. At 64 GB/s that is about 5 µs. The layer before it reads about 0.9 GB of weights per GPU at four-way tensor parallelism (computed, BF16, assuming routing spreads tokens evenly over experts), which takes far longer, so the rows arrive in time.

A single hash head has 16 million rows and 16 million distinct n-grams occur. Roughly what fraction of n-grams share their row with another? How does V4.1 cope with this?

About 1−e−1≈63%1 - e^{-1} \approx 63\%. Every row is still shared, but each n-gram reads eight rows per order hashed by different primes, so two n-grams share at most two of them unless their 63-bit hashes are equal. The projection sees a nearly unique combination of rows, and the gate can turn down a value that doesn’t fit the context.

Why does a product-key memory work best in the middle of the network while Engram works best at layer 2?

A product-key memory computes its query from the hidden state, so its lookup is only as good as the context the hidden state has gathered. Engram’s address comes from the tokens alone; only its gate needs context, and one layer of attention is enough for that, while an early position lets the lookup replace the work early layers do to assemble short patterns.

In the map

Methods

Taxonomy nodes

Further reading

  • Cheng et al. (DeepSeek), Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models (2026), arXiv:2601.07372: the allocation sweep, the ablations and the offload measurement.
  • DeepSeek-V4.1 technical report, §2.4.2, §2.5 and §3.1.3: the shipped configuration, the Sinkhorn-balanced optimizer and the training-side sharding.
  • On the Design of Qwen3.8-Next Architecture (2026), arXiv:2608.30320, §2.3 and §3.1: placement, the two vocabulary-scaling tables and the optimizer assignment.
  • SGLang’s Qwen3.8-Flash-Next day-0 post: the offload kernel and the with/without measurement.
  • Lample et al., Large Memory Layers with Product Keys (2019), arXiv:1907.05242, and Berges et al., Memory Layers at Scale (2024), arXiv:2412.09764: content-addressed memories, their kernels and their placement.
  • Huang et al., Over-Tokenized Transformer (2025), arXiv:2501.16975: hashed n-gram input embeddings, the log-linear loss curve and the MoE result.
  • Yu et al., Scaling Embedding Layers in Language Models (SCONE, 2025), arXiv:2502.01637: frequent n-grams without hashing, and SSD-tier latency measurements.
  • Svenstrup et al., Hash Embeddings (2017), arXiv:1709.03933; Weinberger et al., Feature Hashing (2009), arXiv:0902.2206; Shi et al., Compositional Embeddings Using Complementary Partitions (2020), arXiv:1909.02107.
  • Google’s Gemma 3n developer documentation: PLE caching and the effective-parameter accounting.
↑