Z.ai (Zhipu AI)
GLM-5.2
What GLM-5.2’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- ~753B / ~40B active
- Context
- 1M
- Attention
- sparse (indexer)MLA + DeepSeek Sparse Attention with IndexShare
- Experts
- 256 routed + 1 shared, top-8
- Input
- text
- Licence
- MIT
- Released
- 2026-06-16
Against the consensus recipe
Its documents state 7 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictionused
- Grouped-query attentionnot in its documents
Optimization
- Muonused
- Expert parallelismused
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningused
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modedefault
- Multi-Token Predictionnot in its documents
- FP8used
Checked in its code
Its code shows 7 of the 19 architecture features checked — 4 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_glm_moe_dsa.py, configuration_glm_moe_dsa.py) and config.json at revision cf457fa734, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention | not in its code | not stated | num_attention_heads equals num_key_value_heads (64/64) and attention is MLA-based rather than GQA, so there is no head-sharing group. show evidencenum_attention_heads = 64num_key_value_heads = 64 |
| Multi-head latent attention | in its code | adopted | Attention down-projects hidden states into a shared kv_lora_rank latent (with a decoupled RoPE key) and up-projects per head via… show evidenceAttention down-projects hidden states into a shared kv_lora_rank latent (with a decoupled RoPE key) and up-projects per head via kv_b_proj, DeepSeek-V2/V3 style MLA. kv_lora_rank = 512q_lora_rank = 2048qk_rope_head_dim = 64 |
| Sliding-window attention | not in its code | discussed | No sliding_window key or windowed masking mechanism appears anywhere in the config or code; all layers use the full causal (plus indexer… show the whole verdictNo sliding_window key or windowed masking mechanism appears anywhere in the config or code; all layers use the full causal (plus indexer top-k sparse) mask. |
| Interleaved sliding-window and global attention | not in its code | not stated | There is no layer_types/sliding_window_pattern distinguishing local vs global attention; all 78 layers share the same "indexed_attention"… show evidenceThere is no layer_types/sliding_window_pattern distinguishing local vs global attention; all 78 layers share the same "indexed_attention" type. |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | in its code | adopted | A lightweight GlmMoeDsaIndexer scores past tokens with its own projections and each query attends only to the top index_topk=2048 tokens… show evidenceA lightweight GlmMoeDsaIndexer scores past tokens with its own projections and each query attends only to the top index_topk=2048 tokens it selects (DeepSeek Sparse Attention). index_topk = 2048index_n_heads = 32index_head_dim = 128 |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | No linear-attention or SSM mixer layers exist in this file; every decoder layer uses GlmMoeDsaAttention (MLA with indexer sparsity). |
| Gated DeltaNet layers | not in its code | discussed | No gated delta-rule recurrent mixer is implemented; attention layers are MLA/indexer-sparse, not Gated DeltaNet. |
| Mamba-2 layers | not in its code | not stated | No Mamba-2 SSD state-space blocks are implemented in the provided modeling file. |
| Learnable attention sink | not in its code | not stated | No learned per-head sink logit is added to the softmax denominator anywhere in the attention or indexer code. |
| Gated attention output | not in its code | not stated | The attention output is passed straight to o_proj with no input-dependent sigmoid gate multiplication. show evidence |
| QK normalization | not in its code | not stated | No RMSNorm/LayerNorm is applied to the query or key states before the attention dot product in GlmMoeDsaAttention (only the indexer's… show the whole verdictNo RMSNorm/LayerNorm is applied to the query or key states before the attention dot product in GlmMoeDsaAttention (only the indexer's separate k has a LayerNorm, which is not the main QK dot product). |
| Partial RoPE code only | in its code | not stated | Only the qk_rope_head_dim=64 slice of each 256-dim query/key head is rotated (q_rot/q_pass split), the remaining qk_nope_head_dim=192… show evidenceOnly the qk_rope_head_dim=64 slice of each 256-dim query/key head is rotated (q_rot/q_pass split), the remaining qk_nope_head_dim=192 carries no position signal, as in DeepSeek-style decoupled RoPE. qk_rope_head_dim = 64qk_nope_head_dim = 192 |
| YaRN RoPE scaling | not in its code | not stated | rope_parameters.rope_type is "default" (not yarn), so although yarn_apply_mscale code exists, it is a no-op for this config. show evidencerope_parameters = {"rope_theta": 8000000, "rope_type": "default"} |
| Layers without positional encoding (NoPE) | not in its code | not stated | Every attention layer applies the same decoupled RoPE scheme (rotary applied to the rope slice); there is no subset of softmax attention… show the whole verdictEvery attention layer applies the same decoupled RoPE scheme (rotary applied to the rope slice); there is no subset of softmax attention layers lacking positional encoding entirely. |
| Mixture of experts | in its code | adopted | MoE layers (mlp_layer_types == "sparse") route tokens via GlmMoeDsaTopkRouter to top-8 of 256 routed experts. show evidencen_routed_experts = 256num_experts_per_tok = 8 |
| Shared expert code only | in its code | not stated | GlmMoeDsaMoE adds a shared_experts MLP (sized by n_shared_experts=1) that every token passes through in addition to the routed experts. show evidencen_shared_experts = 1 |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | The router adds a per-expert e_score_correction_bias to scores used only for top-k selection (not for weighting), matching topk_method… show evidenceThe router adds a per-expert e_score_correction_bias to scores used only for top-k selection (not for weighting), matching topk_method "noaux_tc". topk_method = "noaux_tc" |
| Multi-token prediction layers | code unclear | adopted | The config declares num_nextn_predict_layers=1 and weights up to layer 78 are ignored on load, but no MTP module class is present in the… show evidenceThe config declares num_nextn_predict_layers=1 and weights up to layer 78 are ignored on load, but no MTP module class is present in the given modeling file to confirm how it is built. num_nextn_predict_layers = 1 |
| SwiGLU feed-forward code only | in its code | not stated | Both the dense MLP and the MoE experts compute silu(gate_proj(x)) * up_proj(x), i.e. SwiGLU, with hidden_act set to silu. show evidencehidden_act = "silu" |
token mixer 5
softmax attention 1
multi-head latent attention 1
“MLAPO (Multi-head Latent Attention Pre-processing Optimization)”
sparse attention 4
“we adopt DSA (DeepSeek Sparse Attention) [9], a novel architectural innovation that significantly reduces both training and inference costs.”
“every four sparse attention layers”
sparse attention indexer 2
“We propose IndexShare, which reuses the same indexer across every four sparse attention layers, reducing per-token FLOPs by 2.9× at a 1M…”
“Lightning Indexer integrates score calculation, ReLU, and TopK operations into a single kernel”
channel mixer 2
mixture of experts 2
“GLM-5 scales to 256 experts and reduces its layer count to 80 to minimize expert parallelism communication overhead.”
expert routing 1
“We propose a DP-aware routing mechanism to preserve KV cache locality under Data Parallelism (DP) for large-scale MoE inference”
prediction head 2
“Multi-token prediction (MTP) [13;25] increases the performance of base models and acts as draft models for speculative decoding [20].”
“We also improve GLM-5.2’s MTP layer for speculative decoding, increasing the acceptance length by up to 20%”
5 more mentioned or evaluated
gated delta network “Gated DeltaNet (GDN) [54]: A linear attention variant that replaces the quadratic softmax attention computation with a gated linear…”
sliding window attention “Sliding Window Attention (SWA) Interleave: A fixed alternating pattern of full-attention and windowed-attention layers applied uniformly…”
hybrid layer stacking “We introduce a search-based adaptation method that identifies the optimal subset of layers for SWA conversion while retaining full…”
gated delta network “SimpleGDN: A minimalist linearization strategy designed for maximal reuse of pre-trained weights, improving upon GDN for…”
grouped-query attention “In the future, we plan to migrate the inference backend to the training engine and uniformly adopt the Multi-Query Attention (MQA) mode of…”
optimizer 2
“In our experiments with Muon optimizer, we find that MLA with a 576-dimension latent KV-cache cannot match the performance of GQA-8.”
“The method, denoted as Muon Split, enables projection weights for different attention heads to update at different scales.”
learning-rate schedule 2
“including the Muon optimizer, cosine decay, and batch size warmup”
“including the Muon optimizer, cosine decay, and batch size warmup”
training precision 1
“substantially increase throughput via mixed-precision training/rollouts together with MTP and Prefill-Decode (PD) disaggregation”
quantization-aware training 1
“To provide better accuracy at low-precision, we apply INT4 QAT in the SFT stage.”
training stability 1
“we record the failure reason for each sample and exclude samples that fail due to environment collapse”
training parallelism 1
“We utilized a hybrid approach combining Attention Data Parallelism (DP) and MoE Expert Parallelism (EP)”
training runtime 1
“GLM-5 leverages slime’s heartbeat-driven fault-tolerance”
data sourcing 1
“Web Knowledge Graph (WKG) Construction and Question Generation”
data filtering 1
“We apply a three-stage pipeline to balance difficulty and correctness”
synthetic data 1
“we design an agentic data synthesis pipeline comprising three phases”
supervised fine-tuning 1
“rejection sampling fine-tuning and mask fine-tuning”
reinforcement learning algorithm 4
“Our RL algorithm builds upon GRPO [40] and incorporates the IcePop technique [61] to mitigate the training-inference mismatch.”
“Our RL algorithm builds upon GRPO [40] and incorporates the IcePop technique [61] to mitigate the training-inference mismatch.”
“we employ a Direct Double-sided Importance Sampling, which applies a token-level clipping mechanism ( ) to rollout log-probabilities…”
“We adopt the group-wise policy optimization algorithm for RL training”
reward modelling 2
“To supervise the diverse objectives above, we build a hybrid reward system that integrates three complementary types of reward signals…”
“We propose a multi-level reward formulation”
policy distillation 1
“By leveraging on-policy cross-stage distillation as the final refinement, GLM-5 effectively mitigates capability regression while…”
rollout & RL infrastructure 5
“we have engineered a new asynchronous reinforcement learning infrastructure.”
“we developed slime, a novel asynchronous RL infrastructure that substantially improves training throughput and efficiency, enabling more…”
“First, a Token-in-Token-out (TITO) gateway eliminates re-tokenization mismatches by preserving exact action-level correspondence.”
“we present novel asynchronous Agent RL algorithms designed to enhance the quality of autonomous decision-making.”
“we introduce a server-based Multi-Task Rollout Orchestrator for multi-task RL training”
reasoning control 1
“GLM-5 supports controlling the thinking budget through the reasoning_effort parameter, which accepts two levels: max and high. max is the…”
KV cache management 2
“Prefix Cache (extending KV storage to system RAM)”
“Context Management: Features like RadixCache (prefix sharing) and Prefix Cache”
inference quantization 4
“GLM-5 uses FP8 for rollout inference to reduce per-token latency”
“Flex_AWQ_SSZ for scaling calibration”
“Advanced algorithms like QuaRot for outlier suppression”
“we implemented a sophisticated W4A8 mixed-precision quantization strategy”
serving parallelism 2
“We utilized a hybrid approach combining Attention Data Parallelism (DP) and MoE Expert Parallelism (EP)”
“GLM-5, therefore, leverages slime’s Prefill–Decode (PD) disaggregation”
inference scheduling 1
“Asynchronous Scheduling: Within vLLM, we implemented a mechanism to overlap the 'Device-to-Host' (D2H) sampling copies with the…”
inference kernel 1
“For the Sparse Flash Attention kernel, we specifically optimized for GLM-5's sparse patterns”
context management 3
“We use a discard-all strategy as context management for BrowseComp, which is the same as DeepSeek-V3.2, and Kimi K2.5.”
“we combine keep-recent with Discard-all to form a hybrid Hierarchical Context Management strategy”
“we employ a simple Keep-recent-k strategy”
inference engine 6
“vLLM (v0.23.0+)”
“SGLang (v0.5.13.post1+)”
“KTransformers (v0.5.12+)”
“Unsloth (v0.1.47-beta+)”
“vLLM-Ascend”
“xLLM”
training framework 1
“Transformers (v0.5.12+)”
agent product 2
“We evaluate Terminal-Bench 2.1 with Claude Code 2.1.167”
“We run the SWE-Bench Pro suite with OpenHands”
judge 2
“We conduct cross-validation between the Agent-as-a-Judge framework and human experts”
“We use GPT-5.5 (medium) as the judge model”
unfiled 4
“We evaluate Terminal-Bench 2.1 with Terminus-2 framework”
“FlashComm, which splits AllReduce operations to hide communication latency behind computation”
“mask fine-tuning, allowing knowledge acquired during reinforcement learning to be injected back”
“MLAPO fuses 13 small pre-processing operators into one 'super operator'”
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- GLM-5.2 architecture card — LLM Architecture Gallery explainer sebastianraschka.com
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| z.ai/blog/glm-5.2 | official blog | Z.ai | not fetched |
| arxiv.org/abs/2602.15763 | technical report | GLM-5 Team | read |
| github.com/zai-org/GLM-5 | code repo | Z.ai | read |
| huggingface.co/zai-org/GLM-5.2 | model card | Z.ai | read |
| docs.z.ai/guides/llm/glm-5.2 | vendor docs | Z.ai | read |
| interconnects.ai/p/glm-52-is-the-step-change-for-open | third party analysis | Interconnects (Nathan Lambert) | read |
| tech-now.io/en/blogs/glm-5-2 | third party analysis | TechNow | read |
| docs.sglang.io/cookbook/autoregressive/GLM/GLM-5 | vendor docs | SGLang | not fetched |
| lmsys.org/blog/2026-11-02-hicache-distributed-serving | vendor docs | LMSYS | not fetched |