Google DeepMind
Gemma 4 31B
What Gemma 4 31B’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 30.7B dense
- Context
- 256K
- Attention
- sliding/global hybridlocal sliding-window (1024) : global GQA layers, QK-norm, partial RoPE
- Experts
- dense
- Input
- text, image, video
- Licence
- Apache 2.0
- Released
- 2026-04-02
Against the consensus recipe
Its documents state 5 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationused
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictionnot in its documents
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingcore
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modecore
- Multi-Token Predictioncore
- FP8not in its documents
Checked in its code
Its code shows 5 of the 19 architecture features checked — 4 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_gemma4.py, configuration_gemma4.py) and config.json at revision 842da3794e, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention code only | in its code | not stated | Full-attention layers use 32 query heads with 4 global KV heads and sliding layers use 32 query heads with 16 KV heads, both repeated via… show evidenceFull-attention layers use 32 query heads with 4 global KV heads and sliding layers use 32 query heads with 16 KV heads, both repeated via num_key_value_groups/repeat_kv, implementing GQA. text_config.num_attention_heads = 32text_config.num_key_value_heads = 16text_config.num_global_key_value_heads = 4 |
| Multi-head latent attention | not in its code | not stated | No kv_lora_rank/q_lora_rank/latent KV cache mechanism exists; attention uses standard per-head K/V projections sized by num_key_value_heads. |
| Sliding-window attention code only | in its code | not stated | Sliding-attention layers restrict queries with sliding_window=1024 via create_sliding_window_causal_mask and the sliding_window passed… show evidenceSliding-attention layers restrict queries with sliding_window=1024 via create_sliding_window_causal_mask and the sliding_window passed into attention. text_config.sliding_window = 1024 |
| Interleaved sliding-window and global attention | in its code | adopted | layer_types interleaves 5 sliding_attention layers followed by 1 full_attention layer repeatedly across the 60 layers. show evidencetext_config.layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "sliding_atte… |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | not in its code | not stated | No indexer or top-k token selection mechanism (index_topk/index_n_heads) appears anywhere in the code or config. |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | All token-mixing layers are softmax attention (full_attention or sliding_attention); there is no linear-attention or SSM layer type in… show the whole verdictAll token-mixing layers are softmax attention (full_attention or sliding_attention); there is no linear-attention or SSM layer type in layer_types. |
| Gated DeltaNet layers | not in its code | not stated | No gated delta rule / recurrent state update mechanism is implemented in these files. |
| Mamba-2 layers | not in its code | not stated | No Mamba-2/SSD selective state-space block implementation is present in the modeling code. |
| Learnable attention sink | not in its code | not stated | No learned per-head sink logit is added to the attention softmax denominator anywhere in the attention implementation. |
| Gated attention output | not in its code | not stated | Attention output is passed directly to o_proj with no sigmoid gating multiplication (attn_output_gate/use_output_gate not present). show evidence |
| QK normalization code only | in its code | not stated | Text attention normalizes queries and keys per head with Gemma4RMSNorm (q_norm/k_norm) before applying RoPE and computing the dot product. show evidence |
| Partial RoPE code only | in its code | not stated | Full-attention layers' rope_parameters set partial_rotary_factor 0.25, applying rotary embedding to only a quarter of the head dimension… show evidenceFull-attention layers' rope_parameters set partial_rotary_factor 0.25, applying rotary embedding to only a quarter of the head dimension for that layer type. text_config.rope_parameters.full_attention.partial_rotary_factor = 0.25 |
| YaRN RoPE scaling | not in its code | not stated | rope_parameters only declare rope_type 'proportional' and 'default', not YaRN, so no YaRN scaling is configured or applied. |
| Layers without positional encoding (NoPE) | not in its code | not stated | Every unique layer_type (sliding_attention, full_attention) has an associated rope_parameters entry and the rotary embedding is applied… show evidenceEvery unique layer_type (sliding_attention, full_attention) has an associated rope_parameters entry and the rotary embedding is applied unconditionally per layer type, so no attention layer lacks positional encoding. |
| Mixture of experts documents disagree | not in its code | adopted | The MoE block code exists (Gemma4TextRouter/Gemma4TextExperts) but enable_moe_block is false and num_experts/top_k_experts are null, so it… show evidenceThe MoE block code exists (Gemma4TextRouter/Gemma4TextExperts) but enable_moe_block is false and num_experts/top_k_experts are null, so it is switched off for this config. text_config.enable_moe_block = falsetext_config.num_experts = nulltext_config.top_k_experts = null |
| Shared expert | not in its code | not stated | There is no shared-expert mechanism in the code at all (no n_shared_experts config key or always-on expert module), and MoE itself is… show the whole verdictThere is no shared-expert mechanism in the code at all (no n_shared_experts config key or always-on expert module), and MoE itself is disabled. |
| Auxiliary-loss-free load balancing (selection bias) | not in its code | not stated | The router computes softmax probabilities and top-k directly with a per-expert scale applied to weights, not a selection-only bias like… show evidenceThe router computes softmax probabilities and top-k directly with a per-expert scale applied to weights, not a selection-only bias like e_score_correction_bias; no such mechanism exists and MoE is disabled anyway. |
| Multi-token prediction layers | not in its code | not stated | No num_nextn_predict_layers/mtp_num_layers config key or extra next-next-token prediction module exists in the config or code. |
| SwiGLU feed-forward | not in its code | not stated | Text MLP uses hidden_activation 'gelu_pytorch_tanh' (GeGLU-style gated GELU, not SiLU) in act(gate(x))*up(x), so this is GeGLU rather than… show evidenceText MLP uses hidden_activation 'gelu_pytorch_tanh' (GeGLU-style gated GELU, not SiLU) in act(gate(x))*up(x), so this is GeGLU rather than SwiGLU. text_config.hidden_activation = "gelu_pytorch_tanh" |
filed at the root 3
“Rather than adding more layers or parameters to the model, PLE gives each decoder layer its own small embedding for every token.”
“Gemma 4 12B solves these issues by utilizing a single decoder-only transformer”
“while our 31B Dense is maximizing raw quality and provides a powerful foundation for fine-tuning”
token mixer 2
softmax attention 1
global attention 1
“To optimize memory for long contexts, global layers feature unified Keys and Values, and apply Proportional RoPE (p-RoPE).”
hybrid layer stacking 1
“The models employ a hybrid attention mechanism that interleaves local sliding window attention with full global attention, ensuring the…”
channel mixer 2
mixture of experts 2
“Featuring both Dense and Mixture-of-Experts (MoE) architectures, Gemma 4 is well-suited for tasks like text generation, coding, and…”
“By only activating a 4B subset of parameters during inference, the Mixture-of-Experts model runs much faster than its 26B total might…”
positional encoding 3
“To optimize memory for long contexts, global layers feature unified Keys and Values, and apply Proportional RoPE (p-RoPE).”
“Spatial awareness is maintained by adding 2D coordinate-based positional embeddings directly to the patch representations before a final…”
“incorporate both axial 2D-RoPE [Heo et al., 2024] with non-causal attention and 2D absolute positional embeddings”
prediction head 1
“MTP mitigates this inefficiency through speculative decoding, a technique introduced by Google researchers in Fast Inference from…”
multimodal architecture 9
“Gemma 4 12B eliminates these encoders entirely, projecting raw image patches and audio waveforms directly into the LLM's embedding space…”
“Raw audio is segmented into 40ms chunks at 16kHz, resulting in 640-dimensional vectors per chunk. These are projected directly into the…”
“Other Gemma 4 models use dedicated encoders to process multimodal data before passing it to the LLM.”
“As with the vision encoder, we keep weights frozen during pre-training.”
“We replaced Gemma 4’s vision encoder with a lightweight embedding module consisting of a single matrix multiplication, positional…”
“replaces the 550M vision encoder by a single large matmul (35M parameters)”
“The encoder architecture is based on the Universal Speech Model [Zhang et al., 2023, USM], consisting of two downsampling convolution…”
“Our vision encoders support variable aspect ratios (see Figure 2 and Algorithm 1)”
“Gemma 4 supports variable image resolution through a configurable visual token budget, which controls how many tokens are used to…”
quantization-aware training 3
“By simulating quantization during training, QAT minimizes quality loss when the model is compressed.”
“to enable stable inference in fp16, we introduce a scalar scale at each block in order to bound the activation ranges to fit fp16.”
“This release includes QAT checkpoints for the popular Q4_0 quantization format”
training parallelism 2
“For multi-pod training, we perform a data replica reduction over the data center network, using the Pathways approach of Barham et al.…”
“The optimizer state is sharded using an implementation of ZeRO-3 [Ren et al., 2021].”
training runtime 1
“we leverage Slice-Granularity Elasticity [Gemini Team, 2025], which allows continuous training with fewer “slices” of TPU chips when there…”
data sourcing 1
“we extracted all the frames of that segment (at 1 FPS), plus the prompt and the audio from the video”
data filtering 5
“We filter data to decontaminate benchmarks, and to reduce the risk of unwanted or unsafe utterances and the risk of recitation.”
“Rigorous CSAM (Child Sexual Abuse Material) filtering was applied at multiple stages in the data preparation process”
“automated techniques were used to filter out certain personal information and other sensitive data from training sets.”
“Additional methods: Filtering based on content quality and safety in line with our policies.”
“We filter examples that show certain personal information, unsafe or toxic model outputs, mistaken self-identification data, and…”
data mixture & curriculum 1
“Including subsets of data that encourage better in-context attribution, hedging, and refusals to minimize hallucinations also improves…”
tokenization 2
“We use the same tokenizer as Gemini Team [2025] that is, a SentencePiece tokenizer [Kudo and Richardson, 2018] with split digits…”
“PT models output an <eos> token at the end of generation, while IT models output <turn|> at the end of the generation.”
1 mentioned or evaluated
supervised fine-tuning “you can fine-tune Gemma 4 to achieve state-of-the-art performance on your specific tasks”
decoding strategy 4
“We’re releasing Multi-Token Prediction (MTP) drafters for the Gemma 4 family. By using a specialized speculative decoding architecture…”
“Use the following standardized sampling configuration across all use cases: - temperature=1.0 - top_p=0.95 - top_k=64”
“we reduce the decoding overhead by replacing the projection operation to the entire vocabulary by a top-k operation on clusters of tokens”
“The draft models seamlessly utilize the target model's activations and share its KV cache, meaning they don't have to waste time…”
reasoning control 5
“We introduce a thinking mode [OpenAI, 2024] to Gemma 4 models. By outputting a reasoning trace before the response, models demonstrate…”
“Thinking is enabled by including the <|think|>token at the start of the system prompt. To disable thinking, remove the token.”
“Thinking is enabled by including the <|think|>token at the start of the system prompt. To disable thinking, remove the token.”
“Features a 256K token context window, configurable thinking/reasoning mode, native function calling, and multilingual support across 140+…”
“Built-in reasoning mode that lets the model think step-by-step before answering.”
KV cache management 4
“The chart below shows the average price customers are actually paying after prompt caching.”
“We improve memory efficiency by re-using keys as values in the global attention layers (except in E2B and E4B)”
“Combined with KV cache sharing [Shazeer, 2019] and the reuse of keys as values in global layers [Kayyam et al., 2026], these optimizations…”
“leveraging stateless prefix caching in memory to match context history and instantly bypass prefill latency”
inference quantization 7
“Quantization is a key technology to run models on consumer hardware by reducing their memory footprint while also accelerating decode speed.”
“we engineered a custom mobile-quantization schema designed for edge hardware”
“We structured the compressed data to fit the design of mobile accelerators.”
“We focused compression on the model’s vocabulary list and its short-term memory.”
“We pre-calculate these settings during training, which reduces workload on mobile chips and makes responses faster.”
“We heavily compressed (to 2-bit) the specific parts of the model that generate tokens, while keeping the core reasoning layers at higher…”
“Q4_0 quantization: blockwise quantization, often referred to as Q4_0.”
inference scheduling 1
“OpenRouter routes your request to one of them based on the routing mode you pick — Balanced (price + speed), Nitro (fastest), or Exacto…”
context management 2
“Thoughts from previous model turns must not be added before the next user turn begins, with the exception of tool call turns where…”
“Because our audio and vision encoders are not needed in many use cases, you can optimize your memory footprint even further by deploying…”
agentic scaffolding 2
“Native support for structured tool use, enabling agentic workflows.”
“It comes with a secure sandboxed Python execution loop to write, execute, and plot scientific charts inside the chat bubble.”
2 more mentioned or evaluated
inference scheduling “while the 26B mixture-of-experts model presents unique routing challenges at a batch size of 1 on Apple Silicon, processing multiple…”
agentic scaffolding “Features a 256K token context window, configurable thinking/reasoning mode, native function calling, and multilingual support across 140+…”
Not used 1
inference quantization “While PTQ is already effective at preserving quality, our QAT results yield even higher overall quality compared to standard PTQ baselines.”
inference engine 1
“Run Gemma 4 12B as a local, OpenAI-compatible API server using the new litert-lm serve CLI command”
infrastructure service 1
“OpenRouter monitors every provider continuously and automatically retries on the next-best provider when one returns an error.”
filed at the root 1
“all testing was conducted without safety filters to accurately evaluate the model’s inherent capabilities and behaviors”
human & real-world evaluation 1
“A range of automated as well as human evaluations were conducted to help improve model safety.”
1 more mentioned or evaluated
human & real-world evaluation “It's encouraged to perform continuous monitoring (using evaluation metrics, human review)”
other 1
filed at the root 1
“For optimal performance with multimodal inputs, place: Image content before the text in your prompt. Audio content after the text in your…”
unfiled 9
“Freely mix text and images in any order within a single prompt.”
“Gemma 4 introduces native support for the system role, enabling more structured and controllable conversations.”
“OpenRouter routes your request to one of them based on the routing mode you pick — Balanced (price + speed), Nitro (fastest), or Exacto…”
“OpenRouter's API is OpenAI-compatible — most SDKs work by just swapping the base URL.”
“these models activate an effective 2 billion and 4 billion parameter footprint during inference to preserve RAM and battery life”
“For our E2B and E4B edge models, where the final logit calculation becomes a big bottleneck, we even implemented an efficient clustering…”
“For optimal performance with multimodal inputs, place: - Image content before the text in your prompt. - Audio content after the text in…”
“OpenRouter routes your request to one of them based on the routing mode you pick — Balanced (price + speed), Nitro (fastest), or Exacto…”
“OpenRouter routes your request to one of them based on the routing mode you pick — Balanced (price + speed), Nitro (fastest), or Exacto…”
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Gemma 4 31B architecture card — LLM Architecture Gallery explainer sebastianraschka.com
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| blog.google/innovation-and-ai/technology/developers-tools/gemma-4 | official blog | read | |
| arxiv.org/abs/2607.02770 | technical report | Gemma Team, Google DeepMind | read |
| huggingface.co/google/gemma-4-31B-it | model card | Google DeepMind | read |
| huggingface.co/google/gemma-4-26B-A4B-it | model card | Google DeepMind | read |
| ai.google.dev/gemma/docs/core/model_card_4 | vendor docs | Google AI for Developers | read |
| github.com/google-deepmind/gemma | code repo | Google DeepMind | read |
| blog.google/innovation-and-ai/technology/developers-tools/multi-token-p… | official blog | read | |
| blog.google/innovation-and-ai/technology/developers-tools/quantization-… | official blog | read | |
| blog.google/innovation-and-ai/technology/developers-tools/introducing-g… | official blog | read | |
| developers.googleblog.com/gemma-4-12b-the-developer-guide | official blog | Google Developers | read |
| openrouter.ai/google/gemma-4-31b-it | vendor docs | OpenRouter | read |