Open Models Almanac
ModelsGLM-5.3-Flash

Z.ai (Zhipu AI) · usage rank #2

GLM-5.3-Flash

What GLM-5.3-Flash’s own documents say it is built from — every method with the sentence that describes it.

Parameters
320B / 18B active
Context
1M
Attention
linear hybridGated DeltaNet linear layers + MLA sparse (DSA indexer) layers
Experts
288 routed + 1 shared, top-8
Input
text, image, video
Licence
MIT
Released
2026-08-26

Rank by open-weight tokens on OpenRouter, week of 2026-10-26.

Against the consensus recipe

Its documents state 4 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 9 of the 19 architecture features checked — 8 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_glm5_next.py, configuration_glm5_next.py) and config.json at revision 46a050bec1, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention not in its codenot stated

num_attention_heads equals num_key_value_heads (64=64) and the config even enforces this equality, so there is no query-head grouping over…

show evidence

num_attention_heads equals num_key_value_heads (64=64) and the config even enforces this equality, so there is no query-head grouping over fewer KV heads; the attention is MLA, not GQA.

text_config.num_attention_heads = 64text_config.num_key_value_heads = 64
        if self.num_attention_heads != self.num_key_value_heads:
            raise ValueError(
                f"num_attention_heads ({self.num_attention_heads}) must be the same as "
                f"num_key_value_heads ({self.num_key_value_heads})."
            )
configuration_glm5_next.py · L208–212
Multi-head latent attention code only in its codenot stated

Glm5NextTextAttention down-projects KV into a kv_lora_rank latent cached and expands via kv_b_proj per head, DeepSeek-V2/V3 style MLA.

show evidencetext_config.kv_lora_rank = 512text_config.q_lora_rank = 1536
        self.kv_a_proj_with_mqa = nn.Linear(
            self.hidden_size,
            config.kv_lora_rank + config.qk_rope_head_dim,
            bias=config.attention_bias,
        )
        self.kv_a_layernorm = Glm5NextTextRMSNorm(self.kv_lora_rank, eps=config.rms_norm_eps)
        self.kv_b_proj = nn.Linear(
            config.kv_lora_rank,
            self.num_heads * (self.qk_nope_head_dim + self.v_head_dim),
            bias=False,
        )
modeling_glm5_next.py · L1131–1141
Sliding-window attention not in its codenot stated

No sliding_window config key or windowed masking mechanism appears anywhere in the code or config; attention layers are either…

show the whole verdict

No sliding_window config key or windowed masking mechanism appears anywhere in the code or config; attention layers are either linear-attention or full indexed/sparse attention over the whole context.

Interleaved sliding-window and global attention not in its codenot stated

There is no sliding-window layer type at all (layer_types only contains linear_attention and deepseek_sparse_attention/indexed_attention)…

show evidence

There is no sliding-window layer type at all (layer_types only contains linear_attention and deepseek_sparse_attention/indexed_attention), so there is no interleaving of sliding and global attention.

text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "deepseek_sparse…
Indexer-selected sparse attention (DeepSeek Sparse Attention) code only in its codenot stated

Glm5NextTextIndexer scores pooled past tokens and selects top index_topk candidates per query, which are used to build the sparse…

show evidence

Glm5NextTextIndexer scores pooled past tokens and selects top index_topk candidates per query, which are used to build the sparse attention mask in Glm5NextTextAttention (DeepSeek Sparse Attention), active on every indexed_attention layer.

text_config.index_topk = 2048text_config.index_n_heads = 32text_config.index_head_dim = 128
        select_k = min(self.index_topk // self.index_kpool, index_scores.shape[-1])

        # Selection is based on 2 steps
        #   1. The actual scores (selected indices)
        #   2. And removing invalid rows based on padding (selected valid)
        selected = index_scores.topk(select_k, dim=-1).indices
modeling_glm5_next.py · L884–889
        self.skip_topk = config.indexer_types[layer_idx] == "shared"
        self.indexer = None if self.skip_topk else Glm5NextTextIndexer(config, layer_idx)
modeling_glm5_next.py · L1150–1151
Linear-attention or state-space layers alongside full attention code only in its codenot stated

layer_types interleaves linear_attention (KDA) layers with deepseek_sparse_attention (full MLA+DSA) layers in a fixed 3:1 pattern across…

show evidence

layer_types interleaves linear_attention (KDA) layers with deepseek_sparse_attention (full MLA+DSA) layers in a fixed 3:1 pattern across the 45 decoder layers.

text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "deepseek_sparse…
        self.self_attn = (
            Glm5NextTextLinearAttention(config, layer_idx)
            if self.block_type == "linear_attention"
            else Glm5NextTextAttention(config, layer_idx)
        )
modeling_glm5_next.py · L1284–1288
Gated DeltaNet layers code only in its codenot stated

Glm5NextTextLinearAttention implements Kimi Delta Attention (a Gated DeltaNet variant) with a recurrent state updated by a delta-rule…

show evidence

Glm5NextTextLinearAttention implements Kimi Delta Attention (a Gated DeltaNet variant) with a recurrent state updated by a delta-rule write (chunk_kimi_delta_attention/recurrent_kimi_delta_attention) and a decay gate from the forget gate module, active on all linear_attention layers.

text_config.linear_attn_config = {"num_heads": 64, "gate_lower_bound": -5.0, "head_dim": 128, "short_conv_kern…
        last_recurrent_state = last_recurrent_state * g_i
        kv_mem = (last_recurrent_state * k_i[..., None]).sum(dim=-2)
        delta = (v_i - kv_mem) * b_i

        last_recurrent_state = last_recurrent_state + k_i.unsqueeze(-1) * delta.unsqueeze(-2)
modeling_glm5_next.py · L506–510
    """Kimi-style KDA (Kimi Linear Attention) for GLM-5.3-Flash."""
modeling_glm5_next.py · L621
Mamba-2 layers not in its codenot stated

No Mamba-2/SSD mechanism (no mamba_num_heads, ssm_state_size, or per-head scalar decay SSM module) exists; the recurrent mixer implemented…

show the whole verdict

No Mamba-2/SSD mechanism (no mamba_num_heads, ssm_state_size, or per-head scalar decay SSM module) exists; the recurrent mixer implemented is Kimi Delta Attention, not Mamba-2.

Learnable attention sink not in its codenot stated

No learned per-head sink logit or extra softmax denominator column exists anywhere in the attention implementations (MLA/DSA…

show the whole verdict

No learned per-head sink logit or extra softmax denominator column exists anywhere in the attention implementations (MLA/DSA eager_attention_forward has no sink term).

Gated attention output not in its codenot stated

The softmax (MLA/DSA) attention output is passed straight to o_proj with no sigmoid gate; gating by sigmoid only occurs inside the…

show evidence

The softmax (MLA/DSA) attention output is passed straight to o_proj with no sigmoid gate; gating by sigmoid only occurs inside the linear-attention RMSNormGated module (o_norm), which is a normalization gate on the linear-attention branch, not a gated attention-output mechanism on the softmax attention path.

        attn_output = attn_output.reshape(batch_size, seq_length, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights, topk_indices if self.next_skip_topk else None
modeling_glm5_next.py · L1234–1236
QK normalization not in its codenot stated

There is no RMSNorm/LayerNorm applied to the query/key projections inside Glm5NextTextAttention (MLA) before the dot product; only the…

show the whole verdict

There is no RMSNorm/LayerNorm applied to the query/key projections inside Glm5NextTextAttention (MLA) before the dot product; only the vision attention and the DSA indexer's key (k_norm) apply such norms, which do not count for the language model's main attention path.

Partial RoPE not in its codenot stated

qk_rope_head_dim is 0 and mla_use_nope is true, so no rotary position encoding is applied to any part of the query/key heads in this…

show evidence

qk_rope_head_dim is 0 and mla_use_nope is true, so no rotary position encoding is applied to any part of the query/key heads in this config (position_embeddings is explicitly passed as None).

text_config.qk_rope_head_dim = 0text_config.mla_use_nope = true
                # Key change using NoPE
                position_embeddings=None,
modeling_glm5_next.py · L1502–1503
YaRN RoPE scaling not in its codenot stated

No rope_scaling or rope_parameters key of type YaRN appears in text_config, and the text attention path uses NoPE (no rotary applied at…

show the whole verdict

No rope_scaling or rope_parameters key of type YaRN appears in text_config, and the text attention path uses NoPE (no rotary applied at all), so YaRN scaling cannot apply.

Layers without positional encoding (NoPE) not in its codenot stated

All softmax-attention (indexed/DSA) layers uniformly use no positional encoding (qk_rope_head_dim=0, mla_use_nope=true…

show evidence

All softmax-attention (indexed/DSA) layers uniformly use no positional encoding (qk_rope_head_dim=0, mla_use_nope=true, position_embeddings=None for every layer) — this is a uniform NoPE setup rather than some layers having RoPE and others not.

text_config.mla_use_nope = truetext_config.qk_rope_head_dim = 0
                # Key change using NoPE
                position_embeddings=None,
modeling_glm5_next.py · L1502–1503
Mixture of experts in its codeadopted

Glm5NextTextMoE routes tokens via Glm5NextTextTopkRouter (sigmoid scoring, top-k=8 of 288 experts) to Glm5NextTextExperts MLPs on all…

show evidence

Glm5NextTextMoE routes tokens via Glm5NextTextTopkRouter (sigmoid scoring, top-k=8 of 288 experts) to Glm5NextTextExperts MLPs on all mlp_layer_types=='sparse' layers.

text_config.n_routed_experts = 288text_config.num_experts_per_tok = 8
        topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_glm5_next.py · L176–177
Shared expert code only in its codenot stated

Glm5NextTextMoE includes a shared_experts MLP (sized by n_shared_experts=1) added to every token's output alongside the routed experts.

show evidencetext_config.n_shared_experts = 1
        self.shared_experts = Glm5NextTextMLP(
            config=config, intermediate_size=config.moe_intermediate_size * config.n_shared_experts
        )
modeling_glm5_next.py · L195–197
        hidden_states = hidden_states + self.shared_experts(residuals)
modeling_glm5_next.py · L205
Auxiliary-loss-free load balancing (selection bias) code only in its codenot stated

Glm5NextTextTopkRouter adds e_score_correction_bias to the sigmoid scores purely to select top-k experts (scores_for_choice), while the…

show evidence

Glm5NextTextTopkRouter adds e_score_correction_bias to the sigmoid scores purely to select top-k experts (scores_for_choice), while the actual weighting uses the unbiased scores, matching DeepSeek-V3's aux-loss-free balancing; topk_method is noaux_tc.

text_config.topk_method = "noaux_tc"
        scores = router_logits.sigmoid()
        scores_for_choice = scores + self.e_score_correction_bias
modeling_glm5_next.py · L160–161
        topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_glm5_next.py · L176–177
Multi-token prediction layers code only in its codenot stated

num_nextn_predict_layers is 1 and the quantization_config module list declares a full extra layer 45 (eh_proj, enorm, hnorm…

show evidence

num_nextn_predict_layers is 1 and the quantization_config module list declares a full extra layer 45 (eh_proj, enorm, hnorm, shared_head.norm) that modeling code explicitly skips on load via _keys_to_ignore_on_load_unexpected, which matches the MTP-declared-but-skipped pattern.

text_config.num_nextn_predict_layers = 1
    _keys_to_ignore_on_load_unexpected = [r"layers\.45\.", r"layers\.\d+\.shared_head\."]
modeling_glm5_next.py · L1379
SwiGLU feed-forward code only in its codenot stated

hidden_act is silu and Glm5NextTextMLP/Glm5NextTextExperts compute silu(gate) * up before the down projection, satisfying SwiGLU for both…

show evidence

hidden_act is silu and Glm5NextTextMLP/Glm5NextTextExperts compute silu(gate) * up before the down projection, satisfying SwiGLU for both dense and expert FFNs.

text_config.hidden_act = "silu"
        gate = gate.clamp(min=None, max=self.swiglu_limit)
        up = up.clamp(min=-self.swiglu_limit, max=self.swiglu_limit)
        return self.down_proj(self.act_fn(gate) * up)
modeling_glm5_next.py · L101–103

token mixer 4

sparse attention 2

NoPE sparse multi-head latent attention core

“Its 45-layer language model combines KDA linear-attention layers with NoPE sparse MLA layers”

sparse attention indexer 1

IndexPool core

“IndexPool compresses groups of indexer key vectors through weighted pooling to hold down latency and memory.”

linear attention & state space 1

gated delta network 1

Kimi Delta Attention core

“Its 45-layer language model combines KDA linear-attention layers with NoPE sparse MLA layers”

hybrid layer stacking 1

Hybrid Sparse and Linear Attention core

“Its hybrid sparse and linear attention architecture maintains accurate long-context behavior while reducing compute overhead.”

channel mixer 2

mixture of experts 2

Mixture of experts core

“multimodal mixture-of-experts model with approximately 321B total parameters and 18B active parameters per token”

expert routing 1

Top-k expert routing core

“routes each token through 8 of 288 experts”

normalization & residual 1

Manifold-Constrained Hyper-Connections core

“The model adopts Manifold-Constrained Hyper-Connections to improve scaling efficiency.”

multimodal architecture 1

Reusing image tokens for video frames used

“the same image token is reused for video frames (frame spans are delimited by the video start/end tokens)”

tokenization 1

Chat-template image/video placeholder expansion used

“The chat template expands each into its <|begin_of_image|>… / <|begin_of_video|>… placeholder tokens”

decoding strategy 1

Multi-Token Prediction used

“--speculative-config '{"method":"mtp","num_speculative_tokens":5}'”

reasoning control 4

Always-on thinking mode core

“Thinking is always on — the generation prompt opens a thinking block unconditionally.”

Thinking mode default

“GLM-5.3-Flash offers three reasoning effort levels driven by the reasoning_effort field; the default is max”

Chat-template reasoning-effort resolution and injection default

“The chat template resolves effort to max unless reasoning_effort is explicitly "low" or "high" (any other value falls back to max), then…”

clear_thinking chat-template flag optional

“In the chat template for GLM-5.3-Flash, clear_thinking defaults to false if not passed.”

inference quantization 3

FP8 default

“ships native FP8 weights”

MXFP8 block-scale regrouping at load used

“On 950 the block scales are re-grouped to MXFP8 at load (weights stay 1 byte/element).”

FP8 KV-cache quantization optional

“On Blackwell you can add --kv-cache-dtype fp8 to both pools; Hopper does not support FP8 KV cache for this model and must run BF16 KV.”

serving parallelism 4

KDA state and KV-cache layout pinning core

“The KDA conv-state and KV-cache layouts must be pinned identically on both pools”

Prefill-decode disaggregation used

“Split one 8-GPU node into a prefill pool (GPUs 0-3) and a decode pool (GPUs 4-7), bridged by NIXL KV transfer.”

Disaggregated serving across encoding, prefill, and decode used

“using a custom SGLang-based engine that disaggregates encoding, prefill and decoding”

Round-robin routing for prefill-decode disaggregation used

“vllm-router --policy round_robin --vllm-pd-disaggregation”

inference kernel 1

ROCm AITER sparse MLA attention backend used

“--attention-backend ROCM_AITER_MLA_SPARSE”

agentic scaffolding 2

Video understanding and editing optional

“Edit the videos in the footage directory into a 90-second product launch recap.”

Vision-driven UI coding optional

“Based on the page screenshots I provide, fully recreate this product using Next.js and TypeScript.”

infrastructure service 1

Automatic provider failover and retry used

“When an error occurs in an upstream provider, we can recover by routing to another healthy provider, if your request filters allow it.”

benchmark 2

CAD visual reproduction optional

“use build123d to write parametric code that recreates the part.”

Computer-use closed-loop evaluation optional

“Use Computer Use to inspect the currently open application, identify its page structure, core functionality, primary user flows, and…”

unfiled 6

30T-token multimodal pre-training corpus used

“our latest 30T-token multimodal pre-training corpus”

3D scene creation with Blender optional

“Create a complete, editable Blender scene in the current directory depicting a restaurant and bar located on a high floor in a city.”

Financial analysis workflow optional

“Based on the company’s latest financial statements, announcements, and verifiable public information, conduct a comprehensive earnings…”

Game development workflow optional

“develop a playable cooperative cooking game prototype with Godot 4.”

Office deliverables creation optional

“Using the materials in the current directory, create a 15-page business presentation for management.”

Recommended API parameter settings optional

“Recommended Settings: temperature: 1 ,top_p: 0.95 , andreasoning_effort: max .”

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

SourceKindPublisherStatus
z.ai/blog/glm-5.3-flashofficial blogZ.ainot fetched
arxiv.org/abs/2602.15763technical reportGLM-5 Teamread
huggingface.co/zai-org/GLM-5.3-Flashmodel cardZ.airead
github.com/zai-org/GLM-5code repoZ.airead
docs.z.ai/guides/llm/glm-5.3-flashvendor docsZ.airead
recipes.vllm.ai/zai-org/GLM-5.3-Flashvendor docsvLLMread
marktechpost.com/2026/08/26/z-ai-releases-glm-5-3-flash-a-320b-a18b-nat…newsMarkTechPostread
openrouter.ai/z-ai/glm-5.3-flashvendor docsOpenRouterread
↑