Open Models Almanac
Methodsmodel architecture

specific method · filed under model architecture

Per-Layer Embeddings (PLE)

Per-layer embeddings give each decoder layer its own small vector for every token, looked up by token id and multiplied with a narrow projection of the hidden state at the end of that layer. In Gemma 4 E2B the table holds about half the parameters but needs no matrix multiplication, so it can be kept outside the accelerator's memory.

Written for the explainer Conditional memory: lookup tables beside the experts.

Also called per-layer embeddings.

sources
4
model
1
lab adopt it
1
strongest
core

Documented in

Google DeepMind

Evidence

4 quotes from 4 sources, one card per model, each opening with its strongest.

How sources treat it
used 1core 3

Gemma 4 31B

Google DeepMindadopted4 sources

Rather than adding more layers or parameters to the model, PLE gives each decoder layer its own small embedding for every token.

coremodel architecturein Gemma 4Google DeepMind
3 more from this model

The smaller models incorporate Per-Layer Embeddings (PLE) to maximize parameter efficiency in on-device deployments. Rather than adding more layers or parameters to the model, PLE gives each decoder layer its own small embedding for every token.

coremodel architecturein Gemma 4Google DeepMind

E2B and E4B use per-layer embeddings as in Gemma 3n [Gemma Team, 2025b], making them 2.3B and 4.5B effective out of 5B and 8B total parameters respectively.

usedmodel architecturein Gemma 4Google DeepMind

PLE gives each decoder layer its own small embedding for every token.

coremodel architecturein Gemma 4Google DeepMind

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑