Open Models Almanac
Methodsmodel architecturecontext capacity

specific method · filed under model architecture

Engram

Engram adds a table of learned vectors beside the experts: at chosen layers each token reads a few rows addressed by hashes of itself and the preceding tokens, and a scalar gate computed from the hidden state decides how much of them to add to the residual stream. Because the addresses depend only on token ids, the rows can be fetched from host memory while earlier layers run, so the table adds parameters for almost no arithmetic or GPU memory.

Written for the explainer Conditional memory: lookup tables beside the experts.

Also called Engram conditional memory, Engram conditional memory module, sparsely accessed conditional memory.

sources
3
models
2
lab adopt it
1
strongest
core

Documented in

Evidence

3 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
used 1core 1not used 1

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑