GLM-5.3-Flash

Softmax attention layers use DeepSeek-V2-style latent KV compression (kv_lora_rank=512, q_lora_rank=1536) with kv_b_proj up-projection per head, as implemented in Glm5NextTextAttention.

self.kv_a_proj_with_mqa = nn.Linear(
            self.hidden_size,
            config.kv_lora_rank + config.qk_rope_head_dim,
            bias=config.attention_bias,
        )
        self.kv_a_layernorm = Glm5NextTextRMSNorm(self.kv_lora_rank, eps=config.rms_norm_eps)
        self.kv_b_proj = nn.Linear(
            config.kv_lora_rank,
            self.num_heads * (self.qk_nope_head_dim + self.v_head_dim),
            bias=False,
        )

A Glm5NextTextIndexer lightweight indexer scores pooled candidate tokens and each query attends only to the resulting top-k selection (index_topk=2048) in deepseek_sparse_attention layers.

select_k = min(self.index_topk // self.index_kpool, index_scores.shape[-1])

self.wq_b = nn.Linear(self.q_lora_rank, self.n_heads * self.head_dim, bias=False)
        self.wk = nn.Linear(self.hidden_size, self.head_dim, bias=False)

layer_types alternates linear_attention (KDA) layers with deepseek_sparse_attention (MLA+indexer) softmax layers across the 45 decoder layers.

self.self_attn = (
            Glm5NextTextLinearAttention(config, layer_idx)
            if self.block_type == "linear_attention"
            else Glm5NextTextAttention(config, layer_idx)
        )

Glm5NextTextLinearAttention implements Kimi Delta Attention (a gated delta-rule variant) with a forget gate decay and delta-rule state update, used in linear_attention layers.

last_recurrent_state = last_recurrent_state * g_i
        kv_mem = (last_recurrent_state * k_i[..., None]).sum(dim=-2)
        delta = (v_i - kv_mem) * b_i

        last_recurrent_state = last_recurrent_state + k_i.unsqueeze(-1) * delta.unsqueeze(-2)

All softmax-attention (MLA/deepseek_sparse_attention) layers apply no positional encoding at all (qk_rope_head_dim=0, mla_use_nope=true, and the model forward passes position_embeddings=None for every layer), which is a fixed NoPE choice rather than a mix of RoPE/no-RoPE layers, but it does mean the attention layers apply no rotary; since it's uniform across all attention layers (not some layers with and some without), this is effectively a fully NoPE model rather than the interleaved NoPE pattern described.

position_ids=position_ids,
                # Key change using NoPE
                position_embeddings=None,

Glm5NextTextMoE routes each token to num_experts_per_tok=8 of n_routed_experts=288 expert MLPs via a learned sigmoid router (Glm5NextTextTopkRouter) in sparse mlp_layer_types layers.

topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)

Glm5NextTextMoE adds a shared_experts MLP (sized by moe_intermediate_size * n_shared_experts=1) that every token passes through in addition to routed experts.

self.shared_experts = Glm5NextTextMLP(
            config=config, intermediate_size=config.moe_intermediate_size * config.n_shared_experts
        )

The router adds e_score_correction_bias to sigmoid scores solely for top-k expert selection (topk_method noaux_tc), matching DeepSeek-V3's bias-based load balancing.

scores = router_logits.sigmoid()
        scores_for_choice = scores + self.e_score_correction_bias

num_nextn_predict_layers=1 declares an MTP module, and the quantization_config lists model.layers.45.* weights (eh_proj, enorm, hnorm, shared_head) corresponding to the extra next-token-prediction layer beyond the 45 main layers.

Both the dense Glm5NextTextMLP and the MoE Glm5NextTextExperts compute silu(gate(x)) * up(x) (clamped SwiGLU) with hidden_act set to silu.

gate = gate.clamp(min=None, max=self.swiglu_limit)
        up = up.clamp(min=-self.swiglu_limit, max=self.swiglu_limit)
        return self.down_proj(self.act_fn(gate) * up)