Open Models Almanac
ModelsGemma 4 31B

Google DeepMind

Gemma 4 31B

What Gemma 4 31B’s own documents say it is built from — every method with the sentence that describes it.

Parameters
30.7B dense
Context
256K
Attention
sliding/global hybridlocal sliding-window (1024) : global GQA layers, QK-norm, partial RoPE
Experts
dense
Input
text, image, video
Licence
Apache 2.0
Released
2026-04-02

Against the consensus recipe

Its documents state 5 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 5 of the 19 architecture features checked — 4 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_gemma4.py, configuration_gemma4.py) and config.json at revision 842da3794e, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention code only in its codenot stated

Full-attention layers use 32 query heads with 4 global KV heads and sliding layers use 32 query heads with 16 KV heads, both repeated via…

show evidence

Full-attention layers use 32 query heads with 4 global KV heads and sliding layers use 32 query heads with 16 KV heads, both repeated via num_key_value_groups/repeat_kv, implementing GQA.

text_config.num_attention_heads = 32text_config.num_key_value_heads = 16text_config.num_global_key_value_heads = 4
self.num_key_value_groups = config.num_attention_heads // layer_config.num_key_value_heads
modeling_gemma4.py · L1164
    key_states = repeat_kv(key, module.num_key_value_groups)
    value_states = repeat_kv(value, module.num_key_value_groups)
modeling_gemma4.py · L826–827
Multi-head latent attention not in its codenot stated

No kv_lora_rank/q_lora_rank/latent KV cache mechanism exists; attention uses standard per-head K/V projections sized by num_key_value_heads.

Sliding-window attention code only in its codenot stated

Sliding-attention layers restrict queries with sliding_window=1024 via create_sliding_window_causal_mask and the sliding_window passed…

show evidence

Sliding-attention layers restrict queries with sliding_window=1024 via create_sliding_window_causal_mask and the sliding_window passed into attention.

text_config.sliding_window = 1024
self.is_sliding = self.layer_type == "sliding_attention"
        self.sliding_window = config.sliding_window if self.is_sliding else None
modeling_gemma4.py · L1158–1159
"full_attention": create_causal_mask(**mask_kwargs),
                "sliding_attention": create_sliding_window_causal_mask(**mask_kwargs),
modeling_gemma4.py · L1659–1660
Interleaved sliding-window and global attention in its codeadopted

layer_types interleaves 5 sliding_attention layers followed by 1 full_attention layer repeatedly across the 60 layers.

show evidencetext_config.layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "sliding_atte…
causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "sliding_attention": create_sliding_window_causal_mask(**mask_kwargs),
            }
modeling_gemma4.py · L1658–1661
Indexer-selected sparse attention (DeepSeek Sparse Attention) not in its codenot stated

No indexer or top-k token selection mechanism (index_topk/index_n_heads) appears anywhere in the code or config.

Linear-attention or state-space layers alongside full attention not in its codenot stated

All token-mixing layers are softmax attention (full_attention or sliding_attention); there is no linear-attention or SSM layer type in…

show the whole verdict

All token-mixing layers are softmax attention (full_attention or sliding_attention); there is no linear-attention or SSM layer type in layer_types.

Gated DeltaNet layers not in its codenot stated

No gated delta rule / recurrent state update mechanism is implemented in these files.

Mamba-2 layers not in its codenot stated

No Mamba-2/SSD selective state-space block implementation is present in the modeling code.

Learnable attention sink not in its codenot stated

No learned per-head sink logit is added to the attention softmax denominator anywhere in the attention implementation.

Gated attention output not in its codenot stated

Attention output is passed directly to o_proj with no sigmoid gating multiplication (attn_output_gate/use_output_gate not present).

show evidence
attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights
modeling_gemma4.py · L966–968
QK normalization code only in its codenot stated

Text attention normalizes queries and keys per head with Gemma4RMSNorm (q_norm/k_norm) before applying RoPE and computing the dot product.

show evidence
self.q_norm = Gemma4RMSNorm(dim=self.head_dim, eps=config.rms_norm_eps)
modeling_gemma4.py · L1177
self.k_norm = Gemma4RMSNorm(dim=self.head_dim, eps=config.rms_norm_eps)
            self.v_norm = Gemma4RMSNorm(self.head_dim, eps=config.rms_norm_eps, with_scale=False)
modeling_gemma4.py · L1181–1182
Partial RoPE code only in its codenot stated

Full-attention layers' rope_parameters set partial_rotary_factor 0.25, applying rotary embedding to only a quarter of the head dimension…

show evidence

Full-attention layers' rope_parameters set partial_rotary_factor 0.25, applying rotary embedding to only a quarter of the head dimension for that layer type.

text_config.rope_parameters.full_attention.partial_rotary_factor = 0.25
base = config.rope_parameters[layer_type]["rope_theta"]
        dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads
modeling_gemma4.py · L1128–1129
YaRN RoPE scaling not in its codenot stated

rope_parameters only declare rope_type 'proportional' and 'default', not YaRN, so no YaRN scaling is configured or applied.

show evidencetext_config.rope_parameters = {"full_attention": {"partial_rotary_factor": 0.25, "rope_theta": 1000000.0, "…
Layers without positional encoding (NoPE) not in its codenot stated

Every unique layer_type (sliding_attention, full_attention) has an associated rope_parameters entry and the rotary embedding is applied…

show evidence

Every unique layer_type (sliding_attention, full_attention) has an associated rope_parameters entry and the rotary embedding is applied unconditionally per layer type, so no attention layer lacks positional encoding.

for layer_type in self.layer_types:
            rope_params = self.config.rope_parameters[layer_type]
            if rope_params is None:
                continue
modeling_gemma4.py · L1089–1092
Mixture of experts documents disagree not in its codeadopted

The MoE block code exists (Gemma4TextRouter/Gemma4TextExperts) but enable_moe_block is false and num_experts/top_k_experts are null, so it…

show evidence

The MoE block code exists (Gemma4TextRouter/Gemma4TextExperts) but enable_moe_block is false and num_experts/top_k_experts are null, so it is switched off for this config.

text_config.enable_moe_block = falsetext_config.num_experts = nulltext_config.top_k_experts = null
self.enable_moe_block = config.enable_moe_block
        if self.enable_moe_block:
            self.router = Gemma4TextRouter(config)
            self.experts = Gemma4TextExperts(config)
modeling_gemma4.py · L1360–1363
Shared expert not in its codenot stated

There is no shared-expert mechanism in the code at all (no n_shared_experts config key or always-on expert module), and MoE itself is…

show the whole verdict

There is no shared-expert mechanism in the code at all (no n_shared_experts config key or always-on expert module), and MoE itself is disabled.

Auxiliary-loss-free load balancing (selection bias) not in its codenot stated

The router computes softmax probabilities and top-k directly with a per-expert scale applied to weights, not a selection-only bias like…

show evidence

The router computes softmax probabilities and top-k directly with a per-expert scale applied to weights, not a selection-only bias like e_score_correction_bias; no such mechanism exists and MoE is disabled anyway.

top_k_weights, top_k_index = torch.topk(
            router_probabilities,
            k=self.config.top_k_experts,
            dim=-1,
        )  # both [B*S, K]
modeling_gemma4.py · L1324–1328
Multi-token prediction layers not in its codenot stated

No num_nextn_predict_layers/mtp_num_layers config key or extra next-next-token prediction module exists in the config or code.

SwiGLU feed-forward not in its codenot stated

Text MLP uses hidden_activation 'gelu_pytorch_tanh' (GeGLU-style gated GELU, not SiLU) in act(gate(x))*up(x), so this is GeGLU rather than…

show evidence

Text MLP uses hidden_activation 'gelu_pytorch_tanh' (GeGLU-style gated GELU, not SiLU) in act(gate(x))*up(x), so this is GeGLU rather than SwiGLU.

text_config.hidden_activation = "gelu_pytorch_tanh"
def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj
modeling_gemma4.py · L707–709

filed at the root 3

Per-Layer Embeddings (PLE) core

“Rather than adding more layers or parameters to the model, PLE gives each decoder layer its own small embedding for every token.”

Decoder-only Transformer architecture core

“Gemma 4 12B solves these issues by utilizing a single decoder-only transformer”

Dense model architecture used

“while our 31B Dense is maximizing raw quality and provides a powerful foundation for fine-tuning”

token mixer 2

softmax attention 1

global attention 1

Unified Keys and Values core

“To optimize memory for long contexts, global layers feature unified Keys and Values, and apply Proportional RoPE (p-RoPE).”

hybrid layer stacking 1

Hybrid Sliding-Window and Global Attention core

“The models employ a hybrid attention mechanism that interleaves local sliding window attention with full global attention, ensuring the…”

channel mixer 2

mixture of experts 2

Mixture of experts core

“Featuring both Dense and Mixture-of-Experts (MoE) architectures, Gemma 4 is well-suited for tasks like text generation, coding, and…”

Sparse expert activation core

“By only activating a 4B subset of parameters during inference, the Mixture-of-Experts model runs much faster than its 26B total might…”

positional encoding 3

Proportional Rotary Position Embedding used

“To optimize memory for long contexts, global layers feature unified Keys and Values, and apply Proportional RoPE (p-RoPE).”

2D coordinate-based positional embeddings used

“Spatial awareness is maintained by adding 2D coordinate-based positional embeddings directly to the patch representations before a final…”

Axial 2D-RoPE used

“incorporate both axial 2D-RoPE [Heo et al., 2024] with non-causal attention and 2D absolute positional embeddings”

prediction head 1

MTP speculative decoding core

“MTP mitigates this inefficiency through speculative decoding, a technique introduced by Google researchers in Fast Inference from…”

multimodal architecture 9

Encoder-free multimodal architecture core

“Gemma 4 12B eliminates these encoders entirely, projecting raw image patches and audio waveforms directly into the LLM's embedding space…”

Raw audio projection into LLM embedding space used

“Raw audio is segmented into 40ms chunks at 16kHz, resulting in 640-dimensional vectors per chunk. These are projected directly into the…”

Dedicated multimodal encoders before the LLM used

“Other Gemma 4 models use dedicated encoders to process multimodal data before passing it to the LLM.”

Frozen encoders during pre-training used

“As with the vision encoder, we keep weights frozen during pre-training.”

Lightweight vision embedding module used

“We replaced Gemma 4’s vision encoder with a lightweight embedding module consisting of a single matrix multiplication, positional…”

Single-matmul vision projection used

“replaces the 550M vision encoder by a single large matmul (35M parameters)”

Universal Speech Model-based audio encoder used

“The encoder architecture is based on the Universal Speech Model [Zhang et al., 2023, USM], consisting of two downsampling convolution…”

Variable aspect-ratio image handling used

“Our vision encoders support variable aspect ratios (see Figure 2 and Algorithm 1)”

Configurable visual token budget optional

“Gemma 4 supports variable image resolution through a configurable visual token budget, which controls how many tokens are used to…”

quantization-aware training 3

Quantization-Aware Training core

“By simulating quantization during training, QAT minimizes quality loss when the model is compressed.”

Per-Block Scalar Scaling used

“to enable stable inference in fp16, we introduce a scalar scale at each block in order to bound the activation ranges to fit fp16.”

Q4_0 Quantization Format used

“This release includes QAT checkpoints for the popular Q4_0 quantization format”

training parallelism 2

Pathways data-replica reduction used

“For multi-pod training, we perform a data replica reduction over the data center network, using the Pathways approach of Barham et al.…”

ZeRO-3 optimizer-state sharding used

“The optimizer state is sharded using an implementation of ZeRO-3 [Ren et al., 2021].”

training runtime 1

Slice-Granularity Elasticity used

“we leverage Slice-Granularity Elasticity [Gemini Team, 2025], which allows continuous training with fewer “slices” of TPU chips when there…”

data sourcing 1

Video frame extraction at 1 FPS used

“we extracted all the frames of that segment (at 1 FPS), plus the prompt and the audio from the video”

data filtering 5

Benchmark decontamination used

“We filter data to decontaminate benchmarks, and to reduce the risk of unwanted or unsafe utterances and the risk of recitation.”

CSAM filtering used

“Rigorous CSAM (Child Sexual Abuse Material) filtering was applied at multiple stages in the data preparation process”

Sensitive data filtering used

“automated techniques were used to filter out certain personal information and other sensitive data from training sets.”

Content quality and safety filtering used

“Additional methods: Filtering based on content quality and safety in line with our policies.”

post-training data filtering for safety and factuality used

“We filter examples that show certain personal information, unsafe or toxic model outputs, mistaken self-identification data, and…”

data mixture & curriculum 1

In-context attribution, hedging, and refusal data subsets used

“Including subsets of data that encourage better in-context attribution, hedging, and refusals to minimize hallucinations also improves…”

tokenization 2

SentencePiece used

“We use the same tokenizer as Gemini Team [2025] that is, a SentencePiece tokenizer [Kudo and Richardson, 2018] with split digits…”

Separate end tokens for pretraining and instruction-tuned models used

“PT models output an <eos> token at the end of generation, while IT models output <turn|> at the end of the generation.”

1 mentioned or evaluated
Fine-tuning mentioned

supervised fine-tuning “you can fine-tune Gemma 4 to achieve state-of-the-art performance on your specific tasks”

decoding strategy 4

Multi-Token Prediction core

“We’re releasing Multi-Token Prediction (MTP) drafters for the Gemma 4 family. By using a specialized speculative decoding architecture…”

Standardized sampling configuration default

“Use the following standardized sampling configuration across all use cases: - temperature=1.0 - top_p=0.95 - top_k=64”

Clustered top-K projection used

“we reduce the decoding overhead by replacing the projection operation to the entire vocabulary by a top-k operation on clusters of tokens”

Drafter-target KV-cache sharing used

“The draft models seamlessly utilize the target model's activations and share its KV cache, meaning they don't have to waste time…”

reasoning control 5

Thinking mode core

“We introduce a thinking mode [OpenAI, 2024] to Gemma 4 models. By outputting a reasoning trace before the response, models demonstrate…”

control-token-enabled thinking mode core

“Thinking is enabled by including the <|think|>token at the start of the system prompt. To disable thinking, remove the token.”

Configurable thinking mode controlled by the <|think|> token optional

“Thinking is enabled by including the <|think|>token at the start of the system prompt. To disable thinking, remove the token.”

configurable thinking/reasoning mode for LLM inference optional

“Features a 256K token context window, configurable thinking/reasoning mode, native function calling, and multilingual support across 140+…”

Step-by-step thinking mode optional

“Built-in reasoning mode that lets the model think step-by-step before answering.”

KV cache management 4

Prompt caching used

“The chart below shows the average price customers are actually paying after prompt caching.”

Key-as-value reuse in global attention layers used

“We improve memory efficiency by re-using keys as values in the global attention layers (except in E2B and E4B)”

KV cache sharing used

“Combined with KV cache sharing [Shazeer, 2019] and the reuse of keys as values in global layers [Kayyam et al., 2026], these optimizations…”

Stateless in-memory prefix caching used

“leveraging stateless prefix caching in memory to match context history and instantly bypass prefill latency”

inference quantization 7

Quantization core

“Quantization is a key technology to run models on consumer hardware by reducing their memory footprint while also accelerating decode speed.”

Mobile quantization core

“we engineered a custom mobile-quantization schema designed for edge hardware”

Channel-wise quantization used

“We structured the compressed data to fit the design of mobile accelerators.”

Embedding and KV-cache quantization used

“We focused compression on the model’s vocabulary list and its short-term memory.”

Static activation quantization used

“We pre-calculate these settings during training, which reduces workload on mobile chips and makes responses faster.”

Targeted 2-bit quantization used

“We heavily compressed (to 2-bit) the specific parts of the model that generate tokens, while keeping the core reasoning layers at higher…”

Q4_0 optional

“Q4_0 quantization: blockwise quantization, often referred to as Q4_0.”

inference scheduling 1

Exacto routing optional

“OpenRouter routes your request to one of them based on the routing mode you pick — Balanced (price + speed), Nitro (fastest), or Exacto…”

context management 2

Excluding prior thinking from multi-turn history default

“Thoughts from previous model turns must not be added before the next user turn begins, with the exception of tool call turns where…”

Modality-specific deployment optional

“Because our audio and vision encoders are not needed in many use cases, you can optimize your memory footprint even further by deploying…”

agentic scaffolding 2

Function calling core

“Native support for structured tool use, enabling agentic workflows.”

Sandboxed Python execution loop used

“It comes with a secure sandboxed Python execution loop to write, execute, and plot scientific charts inside the chat bubble.”

2 more mentioned or evaluated
Concurrent request batching evaluated

inference scheduling “while the 26B mixture-of-experts model presents unique routing challenges at a batch size of 1 on Apple Silicon, processing multiple…”

native function calling mentioned

agentic scaffolding “Features a 256K token context window, configurable thinking/reasoning mode, native function calling, and multilingual support across 140+…”

Not used 1

Post-training quantization not used

inference quantization “While PTQ is already effective at preserving quality, our QAT results yield even higher overall quality compared to standard PTQ baselines.”

inference engine 1

litert-lm serve used

“Run Gemma 4 12B as a local, OpenAI-compatible API server using the new litert-lm serve CLI command”

infrastructure service 1

Automatic provider failover and retry core

“OpenRouter monitors every provider continuously and automatically retries on the next-best provider when one returns an error.”

filed at the root 1

Evaluation without safety filters used

“all testing was conducted without safety filters to accurately evaluate the model’s inherent capabilities and behaviors”

human & real-world evaluation 1

Human evaluation used

“A range of automated as well as human evaluations were conducted to help improve model safety.”

1 more mentioned or evaluated
Continuous monitoring mentioned

human & real-world evaluation “It's encouraged to perform continuous monitoring (using evaluation metrics, human review)”

filed at the root 1

Prompt modality ordering used

“For optimal performance with multimodal inputs, place: Image content before the text in your prompt. Audio content after the text in your…”

unfiled 9

Interleaved multimodal input core

“Freely mix text and images in any order within a single prompt.”

Native system prompt support core

“Gemma 4 introduces native support for the system role, enabling more structured and controllable conversations.”

provider request routing by selected routing mode core

“OpenRouter routes your request to one of them based on the routing mode you pick — Balanced (price + speed), Nitro (fastest), or Exacto…”

OpenAI-compatible API default

“OpenRouter's API is OpenAI-compatible — most SDKs work by just swapping the base URL.”

Effective parameter activation for edge inference used

“these models activate an effective 2 billion and 4 billion parameter footprint during inference to preserve RAM and battery life”

efficient clustering in the embedder used

“For our E2B and E4B edge models, where the final logit calculation becomes a big bottleneck, we even implemented an efficient clustering…”

multimodal prompt content ordering used

“For optimal performance with multimodal inputs, place: - Image content before the text in your prompt. - Audio content after the text in…”

Balanced routing: price + speed per request optional

“OpenRouter routes your request to one of them based on the routing mode you pick — Balanced (price + speed), Nitro (fastest), or Exacto…”

Nitro routing: fastest provider per request optional

“OpenRouter routes your request to one of them based on the routing mode you pick — Balanced (price + speed), Nitro (fastest), or Exacto…”

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

↑