DeepSeek-V4.1-Flash
DeepSeekwe integrate the Engram ... conditional memory module to strengthen model capabilities
1 more from this model
Engram conditional memory (196B parameters, sparsely accessed via token-based lookup)
specific method · filed under model architecture
Engram adds a table of learned vectors beside the experts: at chosen layers each token reads a few rows addressed by hashes of itself and the preceding tokens, and a scalar gate computed from the hidden state decides how much of them to add to the residual stream. Because the addresses depend only on token ids, the rows can be fetched from host memory while earlier layers run, so the table adds parameters for almost no arithmetic or GPU memory.
Written for the explainer Conditional memory: lookup tables beside the experts.
Also called Engram conditional memory, Engram conditional memory module, sparsely accessed conditional memory.
3 quotes from 3 sources, one card per model, each opening with its strongest.
we integrate the Engram ... conditional memory module to strengthen model capabilities
Engram conditional memory (196B parameters, sparsely accessed via token-based lookup)
Engram conditional memory is absent from V4
Picked by hand, not extracted: where to read more, not evidence for anything on this page.