ModelsGemma 4 31B

Google DeepMind · released 2026-04-02

Gemma 4 31B

What Gemma 4 31B’s own documents say it is built from — every method with the sentence that describes it.

Curator’s noteFlagship of the Gemma 4 family: 30.7B dense multimodal, 256K context, configurable thinking mode, Apache 2.0. The family also ships E2B/E4B/12B dense and a 26B-A4B MoE (3.8B active); the sibling model cards and the family technical report are listed as sources rather than as separate entries, since the method content is shared.

Against the consensus recipe

Its documents state 3 of the 15 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something the documents do not say, not something the model lacks.

Data curation
- Agentic data synthesis pipelinenot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- KL alignment lossnot in its documents
Model architecture
- Mixture of Expertscore
- Hybrid Attentioncore
- Multi-Token Predictionnot in its documents
Optimization
- Muonnot in its documents
- Expert Parallelismnot in its documents
- Cosine decaynot in its documents
Post-training
- Group Relative Policy Optimizationnot in its documents
- Supervised fine-tuningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Configurable reasoning effortnot in its documents
- Speculative decodingcore
- FP8 KV-cache quantizationnot in its documents

Checked in its code

Its code shows 5 of the 19 architecture features checked — 5 of them are not stated in its documents.

Documents say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Feature	Code	Its documents	Evidence
Grouped-query attention code only	in its code	not stated	32 query heads share 16 (or 4 for global layers) key/value heads via repeat_kv grouping in Gemma4TextAttention. text_config.num_attention_heads = 32 text_config.num_key_value_heads = 16 text_config.num_global_key_value_heads = 4 `self.num_key_value_groups = config.num_attention_heads // layer_config.num_key_value_heads` modeling_gemma4.py · L1176 `key_states = repeat_kv(key, module.num_key_value_groups) value_states = repeat_kv(value, module.num_key_value_groups)` modeling_gemma4.py · L828–829
Multi-head latent attention	not in its code	not stated	No low-rank KV latent projection (kv_lora_rank/q_lora_rank) exists in the text attention implementation; attention uses standard per-head k/v projections.
Sliding-window attention code only	in its code	not stated	Most layers are typed 'sliding_attention' with sliding_window=1024 enforced through the sliding-window causal mask and passed to the attention interface. text_config.sliding_window = 1024 text_config.layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "sliding_atte… `self.is_sliding = self.layer_type == "sliding_attention" self.sliding_window = config.sliding_window if self.is_sliding else None` modeling_gemma4.py · L1170–1171
Interleaved sliding-window and global attention code only	in its code	not stated	layer_types interleaves five sliding_attention layers followed by one full_attention layer throughout the 60 layers, with separate masks built for each type. text_config.layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "sliding_atte… `causal_mask_mapping = { "full_attention": create_causal_mask(mask_kwargs), "sliding_attention": create_sliding_window_causal_mask(mask_kwargs), }` modeling_gemma4.py · L1672–1675
Indexer-selected sparse attention (DeepSeek Sparse Attention)	not in its code	not stated	No indexer scoring mechanism or top-k token selection code exists in this file for the language model.
Linear-attention or state-space layers alongside full attention	not in its code	not stated	No linear-attention, Gated DeltaNet, Kimi Delta Attention, or Mamba layer types appear; layer_types only contains sliding_attention and full_attention (softmax attention). text_config.layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "sliding_atte…
Gated DeltaNet layers	not in its code	not stated	No gated delta-rule recurrent mixer is implemented; the model uses only softmax attention layer types.
Mamba-2 layers	not in its code	not stated	No Mamba-2/SSD state-space block implementation is present in the code.
Learnable attention sink	not in its code	not stated	No learned per-head sink logit is added to the attention softmax denominator anywhere in eager_attention_forward or Gemma4TextAttention.
Gated attention output	not in its code	not stated	The attention output goes straight to o_proj with no sigmoid gate multiplication applied to attn_output before the output projection.
QK normalization code only	in its code	not stated	Queries and keys are each passed through a per-head RMSNorm (q_norm/k_norm) before the rotary embedding and dot product. `query_states = self.q_proj(hidden_states).view(hidden_shape) query_states = self.q_norm(query_states) query_states = apply_rotary_pos_emb(query_states, cos, sin, unsqueeze_dim=2)` modeling_gemma4.py · L1228–1230 `key_states = self.k_norm(key_states) key_states = apply_rotary_pos_emb(key_states, cos, sin, unsqueeze_dim=2)` modeling_gemma4.py · L1245–1246
Partial RoPE code only	in its code	not stated	Full-attention layers use a partial_rotary_factor of 0.25 in rope_parameters, applying RoPE to only a quarter of the head dimension. text_config.rope_parameters.full_attention.partial_rotary_factor = 0.25 `"full_attention": {"rope_type": "proportional", "partial_rotary_factor": 0.25, "rope_theta": 1_000_000.0},` configuration_gemma4.py · L205
YaRN RoPE scaling	not in its code	not stated	rope_parameters specify rope_type 'proportional' and 'default', not YaRN, and no YaRN scaling code path is triggered by this config. text_config.rope_parameters = {"full_attention": {"partial_rotary_factor": 0.25, "rope_theta": 1000000.0, "…
Layers without positional encoding (NoPE)	not in its code	not stated	Every layer type (full_attention and sliding_attention) has an entry in rope_parameters and receives rotary embeddings; no attention layer type is configured to skip RoPE entirely. text_config.rope_parameters = {"full_attention": {"partial_rotary_factor": 0.25, "rope_theta": 1000000.0, "… `for layer_type in self.layer_types: rope_params = self.config.rope_parameters[layer_type] if rope_params is None: continue` modeling_gemma4.py · L1092–1095
Mixture of experts documents disagree	not in its code	core	MoE code (router + experts) exists but enable_moe_block is false and num_experts/top_k_experts are null in this config, so no routing occurs. text_config.enable_moe_block = false text_config.num_experts = null text_config.top_k_experts = null `self.enable_moe_block = config.enable_moe_block if self.enable_moe_block: self.router = Gemma4TextRouter(config) self.experts = Gemma4TextExperts(config)` modeling_gemma4.py · L1376–1379
Shared expert	not in its code	not stated	No shared-expert mechanism (always-applied expert MLP) exists in the code; only routed experts (Gemma4TextExperts) are implemented, and MoE itself is disabled anyway. text_config.enable_moe_block = false
Auxiliary-loss-free load balancing (selection bias)	not in its code	not stated	The router uses a plain softmax + topk with a per-expert output scale, not a selection-only bias like e_score_correction_bias, and no such config key is present. `top_k_weights, top_k_index = torch.topk( router_probabilities, k=self.config.top_k_experts, dim=-1, ) # both [B*S, K]` modeling_gemma4.py · L1340–1344
Multi-token prediction layers	not in its code	not stated	No MTP module declarations or config keys (num_nextn_predict_layers, mtp_num_layers, etc.) exist in this config or code.
SwiGLU feed-forward	not in its code	not stated	The MLP computes act(gate(x)) * up(x), but hidden_activation is 'gelu_pytorch_tanh' (GeGLU), not SiLU-based SwiGLU. text_config.hidden_activation = "gelu_pytorch_tanh" `def forward(self, x): down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x)) return down_proj` modeling_gemma4.py · L702–704

Document	Kind	Publisher	Status
blog.google/innovation-and-ai/technology/developers-tools/gemma-4	official blog	Google	read
arxiv.org/abs/2607.02770	technical report	Gemma Team, Google DeepMind	read
huggingface.co/google/gemma-4-31B-it	model card	Google DeepMind	read
huggingface.co/google/gemma-4-26B-A4B-it	model card	Google DeepMind	read
ai.google.dev/gemma/docs/core/model_card_4	vendor docs	Google AI for Developers	read
github.com/google-deepmind/gemma	code repo	Google DeepMind	read
blog.google/innovation-and-ai/technology/developers-tools/multi-token-p…	official blog	Google	read
blog.google/innovation-and-ai/technology/developers-tools/quantization-…	official blog	Google	read
blog.google/innovation-and-ai/technology/developers-tools/introducing-g…	official blog	Google	read
developers.googleblog.com/gemma-4-12b-the-developer-guide	official blog	Google Developers	read
openrouter.ai/google/gemma-4-31b-it	vendor docs	OpenRouter	read

Gemma 4 31B

Its documents state 3 of the 15 methods the field agrees on.

Data curation

Training objective

Model architecture

Optimization

Post-training

Inference & serving

Its code shows 5 of the 19 architecture features checked — 5 of them are not stated in its documents.

model architecture 19

filed at the root 3

token mixer 2

softmax attention 1

global attention 1

hybrid layer stacking 1

channel mixer 2

mixture of experts 2

positional encoding 3

multimodal architecture 9

optimization 6

quantization-aware training 3

training parallelism 2

training runtime 1

data curation 9

data sourcing 1

data filtering 5

data mixture & curriculum 1

tokenization 2

post-training 1

supervised fine-tuning 1

inference & serving 28

decoding strategy 5

reasoning control 4

KV cache management 4

inference quantization 9

inference scheduling 2

context management 2

agentic scaffolding 2

software implementation 2

inference engine 1

infrastructure service 1

evaluation 3

filed at the root 1

human & real-world evaluation 2

other 1

filed at the root 1

unfiled 9

Further reading

Sources