PublishersGemma Team, Google DeepMind

Gemma Team, Google DeepMind

1 document read from this publisher.

Gemma 4 Technical Report

technical report · 2026-07-02 · 29 techniques

Mixture of Experts Hybrid Attention Multi-Token Prediction Default thinking mode Quantization-Aware Training Encoder-free multimodal architecture Per-Layer Embeddings (PLE)2D rotary position embedding Evaluation without safety filters Raw audio projection into the LLM embedding space 2D coordinate-based positional embeddings Attribution, hedging, and refusal data subsets Data Replica Reduction over the Data Center Network Frozen encoders during pre-training Key-value reuse in global attention layers KV-cache sharing Mobile quantization Per-Block Scalar Scaling Post-training data filtering for safety and factuality Pre-training data filtering for decontamination and safety Q4_0 quantization SentencePiece tokenizer Separate PT and IT end tokens Single-matrix-multiplication vision projection Slice-Granularity Elasticity Top-k over token clusters Universal Speech Model (USM)-based audio encoder Variable aspect-ratio image handling ZeRO-3 Optimizer State Sharding