Qwen3.8-Flash-Next

Full/indexed attention layers project 24 query heads down to 2 KV heads and repeat_kv expands them for the dot product.

self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads

layer_types alternates linear_attention (GatedDeltaNet) layers with full/indexed softmax attention layers every 4th layer.

self.layer_type = config.layer_types[layer_idx]
        if self.layer_type == "linear_attention":
            self.linear_attn = Qwen4ExpTextGatedDeltaNet(config, layer_idx)
        else:
            self.self_attn = Qwen4ExpTextAttention(config, layer_idx)

Qwen4ExpTextGatedDeltaNet implements the gated delta rule with a decay gate (A_log/dt_bias) and delta-rule state update, used on linear_attention layers.

g = -self.A_log.float().exp() * F.softplus(a.float() + self.dt_bias)

last_recurrent_state = last_recurrent_state * chunk_decay[:, :, i] + key[:, :, i].transpose(-1, -2) @ v_new

Attention output is multiplied by a sigmoid gate derived from a chunk of the q_proj output before the output projection.

attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = attn_output * torch.sigmoid(gate)

Qwen4ExpTextAttention applies per-head RMSNorm to queries and keys before RoPE and the attention dot product.

query_states = self.q_norm(query_states.view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)

partial_rotary_factor of 0.25 means RoPE is applied only to a quarter of the 256-dim head, the rest carrying no position signal via rotate/nope split.

q_rope, q_nope = q[..., :rotary_dim], q[..., rotary_dim:]

Qwen4ExpTextSparseMoeBlock routes each token via a top-k router (Qwen4ExpTextTopKRouter) to a subset of the 512 routed experts.

router_top_value, router_indices = torch.topk(router_probs, self.top_k, dim=-1)  # (seq_len, top_k)

Each MoE block also runs a shared_expert MLP (sized by shared_expert_intermediate_size=640) through every token, gated and added to the routed output.

shared_expert_output = F.sigmoid(self.shared_expert_gate(hidden_states_reshaped)) * shared_expert_output

The config declares an mtp block with num_hidden_layers=1 (and mtp_num_hidden_layers=1), a nonzero count of extra next-token-prediction layers, even though the given modeling file skips loading mtp weights.

_keys_to_ignore_on_load_unexpected = [r"^mtp.*"]

Both the dense MLP and expert MLPs compute silu(gate_proj(x)) * up_proj(x), i.e. SwiGLU, matching hidden_act=silu.

def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj