Z.ai (Zhipu AI) · usage rank #2
GLM-5.3-Flash
What GLM-5.3-Flash’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 320B / 18B active
- Context
- 1M
- Attention
- linear hybridGated DeltaNet linear layers + MLA sparse (DSA indexer) layers
- Experts
- 288 routed + 1 shared, top-8
- Input
- text, image, video
- Licence
- MIT
- Released
- 2026-08-26
Rank by open-weight tokens on OpenRouter, week of 2026-10-26.
Against the consensus recipe
Its documents state 4 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictionnot in its documents
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modedefault
- Multi-Token Predictionused
- FP8default
Checked in its code
Its code shows 9 of the 19 architecture features checked — 8 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_glm5_next.py, configuration_glm5_next.py) and config.json at revision 46a050bec1, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention | not in its code | not stated | num_attention_heads equals num_key_value_heads (64=64) and the config even enforces this equality, so there is no query-head grouping over… show evidencenum_attention_heads equals num_key_value_heads (64=64) and the config even enforces this equality, so there is no query-head grouping over fewer KV heads; the attention is MLA, not GQA. text_config.num_attention_heads = 64text_config.num_key_value_heads = 64 |
| Multi-head latent attention code only | in its code | not stated | Glm5NextTextAttention down-projects KV into a kv_lora_rank latent cached and expands via kv_b_proj per head, DeepSeek-V2/V3 style MLA. show evidencetext_config.kv_lora_rank = 512text_config.q_lora_rank = 1536 |
| Sliding-window attention | not in its code | not stated | No sliding_window config key or windowed masking mechanism appears anywhere in the code or config; attention layers are either… show the whole verdictNo sliding_window config key or windowed masking mechanism appears anywhere in the code or config; attention layers are either linear-attention or full indexed/sparse attention over the whole context. |
| Interleaved sliding-window and global attention | not in its code | not stated | There is no sliding-window layer type at all (layer_types only contains linear_attention and deepseek_sparse_attention/indexed_attention)… show evidenceThere is no sliding-window layer type at all (layer_types only contains linear_attention and deepseek_sparse_attention/indexed_attention), so there is no interleaving of sliding and global attention. text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "deepseek_sparse… |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) code only | in its code | not stated | Glm5NextTextIndexer scores pooled past tokens and selects top index_topk candidates per query, which are used to build the sparse… show evidenceGlm5NextTextIndexer scores pooled past tokens and selects top index_topk candidates per query, which are used to build the sparse attention mask in Glm5NextTextAttention (DeepSeek Sparse Attention), active on every indexed_attention layer. text_config.index_topk = 2048text_config.index_n_heads = 32text_config.index_head_dim = 128 |
| Linear-attention or state-space layers alongside full attention code only | in its code | not stated | layer_types interleaves linear_attention (KDA) layers with deepseek_sparse_attention (full MLA+DSA) layers in a fixed 3:1 pattern across… show evidencelayer_types interleaves linear_attention (KDA) layers with deepseek_sparse_attention (full MLA+DSA) layers in a fixed 3:1 pattern across the 45 decoder layers. text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "deepseek_sparse… |
| Gated DeltaNet layers code only | in its code | not stated | Glm5NextTextLinearAttention implements Kimi Delta Attention (a Gated DeltaNet variant) with a recurrent state updated by a delta-rule… show evidenceGlm5NextTextLinearAttention implements Kimi Delta Attention (a Gated DeltaNet variant) with a recurrent state updated by a delta-rule write (chunk_kimi_delta_attention/recurrent_kimi_delta_attention) and a decay gate from the forget gate module, active on all linear_attention layers. text_config.linear_attn_config = {"num_heads": 64, "gate_lower_bound": -5.0, "head_dim": 128, "short_conv_kern… |
| Mamba-2 layers | not in its code | not stated | No Mamba-2/SSD mechanism (no mamba_num_heads, ssm_state_size, or per-head scalar decay SSM module) exists; the recurrent mixer implemented… show the whole verdictNo Mamba-2/SSD mechanism (no mamba_num_heads, ssm_state_size, or per-head scalar decay SSM module) exists; the recurrent mixer implemented is Kimi Delta Attention, not Mamba-2. |
| Learnable attention sink | not in its code | not stated | No learned per-head sink logit or extra softmax denominator column exists anywhere in the attention implementations (MLA/DSA… show the whole verdictNo learned per-head sink logit or extra softmax denominator column exists anywhere in the attention implementations (MLA/DSA eager_attention_forward has no sink term). |
| Gated attention output | not in its code | not stated | The softmax (MLA/DSA) attention output is passed straight to o_proj with no sigmoid gate; gating by sigmoid only occurs inside the… show evidenceThe softmax (MLA/DSA) attention output is passed straight to o_proj with no sigmoid gate; gating by sigmoid only occurs inside the linear-attention RMSNormGated module (o_norm), which is a normalization gate on the linear-attention branch, not a gated attention-output mechanism on the softmax attention path. |
| QK normalization | not in its code | not stated | There is no RMSNorm/LayerNorm applied to the query/key projections inside Glm5NextTextAttention (MLA) before the dot product; only the… show the whole verdictThere is no RMSNorm/LayerNorm applied to the query/key projections inside Glm5NextTextAttention (MLA) before the dot product; only the vision attention and the DSA indexer's key (k_norm) apply such norms, which do not count for the language model's main attention path. |
| Partial RoPE | not in its code | not stated | qk_rope_head_dim is 0 and mla_use_nope is true, so no rotary position encoding is applied to any part of the query/key heads in this… show evidenceqk_rope_head_dim is 0 and mla_use_nope is true, so no rotary position encoding is applied to any part of the query/key heads in this config (position_embeddings is explicitly passed as None). text_config.qk_rope_head_dim = 0text_config.mla_use_nope = true |
| YaRN RoPE scaling | not in its code | not stated | No rope_scaling or rope_parameters key of type YaRN appears in text_config, and the text attention path uses NoPE (no rotary applied at… show the whole verdictNo rope_scaling or rope_parameters key of type YaRN appears in text_config, and the text attention path uses NoPE (no rotary applied at all), so YaRN scaling cannot apply. |
| Layers without positional encoding (NoPE) | not in its code | not stated | All softmax-attention (indexed/DSA) layers uniformly use no positional encoding (qk_rope_head_dim=0, mla_use_nope=true… show evidenceAll softmax-attention (indexed/DSA) layers uniformly use no positional encoding (qk_rope_head_dim=0, mla_use_nope=true, position_embeddings=None for every layer) — this is a uniform NoPE setup rather than some layers having RoPE and others not. text_config.mla_use_nope = truetext_config.qk_rope_head_dim = 0 |
| Mixture of experts | in its code | adopted | Glm5NextTextMoE routes tokens via Glm5NextTextTopkRouter (sigmoid scoring, top-k=8 of 288 experts) to Glm5NextTextExperts MLPs on all… show evidenceGlm5NextTextMoE routes tokens via Glm5NextTextTopkRouter (sigmoid scoring, top-k=8 of 288 experts) to Glm5NextTextExperts MLPs on all mlp_layer_types=='sparse' layers. text_config.n_routed_experts = 288text_config.num_experts_per_tok = 8 |
| Shared expert code only | in its code | not stated | Glm5NextTextMoE includes a shared_experts MLP (sized by n_shared_experts=1) added to every token's output alongside the routed experts. show evidencetext_config.n_shared_experts = 1 |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | Glm5NextTextTopkRouter adds e_score_correction_bias to the sigmoid scores purely to select top-k experts (scores_for_choice), while the… show evidenceGlm5NextTextTopkRouter adds e_score_correction_bias to the sigmoid scores purely to select top-k experts (scores_for_choice), while the actual weighting uses the unbiased scores, matching DeepSeek-V3's aux-loss-free balancing; topk_method is noaux_tc. text_config.topk_method = "noaux_tc" |
| Multi-token prediction layers code only | in its code | not stated | num_nextn_predict_layers is 1 and the quantization_config module list declares a full extra layer 45 (eh_proj, enorm, hnorm… show evidencenum_nextn_predict_layers is 1 and the quantization_config module list declares a full extra layer 45 (eh_proj, enorm, hnorm, shared_head.norm) that modeling code explicitly skips on load via _keys_to_ignore_on_load_unexpected, which matches the MTP-declared-but-skipped pattern. text_config.num_nextn_predict_layers = 1 |
| SwiGLU feed-forward code only | in its code | not stated | hidden_act is silu and Glm5NextTextMLP/Glm5NextTextExperts compute silu(gate) * up before the down projection, satisfying SwiGLU for both… show evidencehidden_act is silu and Glm5NextTextMLP/Glm5NextTextExperts compute silu(gate) * up before the down projection, satisfying SwiGLU for both dense and expert FFNs. text_config.hidden_act = "silu" |
token mixer 4
sparse attention 2
“Its 45-layer language model combines KDA linear-attention layers with NoPE sparse MLA layers”
sparse attention indexer 1
“IndexPool compresses groups of indexer key vectors through weighted pooling to hold down latency and memory.”
linear attention & state space 1
gated delta network 1
“Its 45-layer language model combines KDA linear-attention layers with NoPE sparse MLA layers”
hybrid layer stacking 1
“Its hybrid sparse and linear attention architecture maintains accurate long-context behavior while reducing compute overhead.”
channel mixer 2
mixture of experts 2
“multimodal mixture-of-experts model with approximately 321B total parameters and 18B active parameters per token”
expert routing 1
“routes each token through 8 of 288 experts”
normalization & residual 1
“The model adopts Manifold-Constrained Hyper-Connections to improve scaling efficiency.”
multimodal architecture 1
“the same image token is reused for video frames (frame spans are delimited by the video start/end tokens)”
tokenization 1
“The chat template expands each into its <|begin_of_image|>… / <|begin_of_video|>… placeholder tokens”
decoding strategy 1
“--speculative-config '{"method":"mtp","num_speculative_tokens":5}'”
reasoning control 4
“Thinking is always on — the generation prompt opens a thinking block unconditionally.”
“GLM-5.3-Flash offers three reasoning effort levels driven by the reasoning_effort field; the default is max”
“The chat template resolves effort to max unless reasoning_effort is explicitly "low" or "high" (any other value falls back to max), then…”
“In the chat template for GLM-5.3-Flash, clear_thinking defaults to false if not passed.”
inference quantization 3
“ships native FP8 weights”
“On 950 the block scales are re-grouped to MXFP8 at load (weights stay 1 byte/element).”
“On Blackwell you can add --kv-cache-dtype fp8 to both pools; Hopper does not support FP8 KV cache for this model and must run BF16 KV.”
serving parallelism 4
“The KDA conv-state and KV-cache layouts must be pinned identically on both pools”
“Split one 8-GPU node into a prefill pool (GPUs 0-3) and a decode pool (GPUs 4-7), bridged by NIXL KV transfer.”
“using a custom SGLang-based engine that disaggregates encoding, prefill and decoding”
“vllm-router --policy round_robin --vllm-pd-disaggregation”
inference kernel 1
“--attention-backend ROCM_AITER_MLA_SPARSE”
agentic scaffolding 2
“Edit the videos in the footage directory into a 90-second product launch recap.”
“Based on the page screenshots I provide, fully recreate this product using Next.js and TypeScript.”
infrastructure service 1
“When an error occurs in an upstream provider, we can recover by routing to another healthy provider, if your request filters allow it.”
benchmark 2
“use build123d to write parametric code that recreates the part.”
“Use Computer Use to inspect the currently open application, identify its page structure, core functionality, primary user flows, and…”
unfiled 6
“our latest 30T-token multimodal pre-training corpus”
“Create a complete, editable Blender scene in the current directory depicting a restaurant and bar located on a high floor in a city.”
“Based on the company’s latest financial statements, announcements, and verifiable public information, conduct a comprehensive earnings…”
“develop a playable cooperative cooking game prototype with Godot 4.”
“Using the materials in the current directory, create a 15-page business presentation for management.”
“Recommended Settings: temperature: 1 ,top_p: 0.95 , andreasoning_effort: max .”
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- GLM-5.3-Flash architecture card — LLM Architecture Gallery explainer sebastianraschka.com
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| z.ai/blog/glm-5.3-flash | official blog | Z.ai | not fetched |
| arxiv.org/abs/2602.15763 | technical report | GLM-5 Team | read |
| huggingface.co/zai-org/GLM-5.3-Flash | model card | Z.ai | read |
| github.com/zai-org/GLM-5 | code repo | Z.ai | read |
| docs.z.ai/guides/llm/glm-5.3-flash | vendor docs | Z.ai | read |
| recipes.vllm.ai/zai-org/GLM-5.3-Flash | vendor docs | vLLM | read |
| marktechpost.com/2026/08/26/z-ai-releases-glm-5-3-flash-a-320b-a18b-nat… | news | MarkTechPost | read |
| openrouter.ai/z-ai/glm-5.3-flash | vendor docs | OpenRouter | read |