DeepSeek-V4-Flash-Vision-Exp

Every attention layer caches and attends only over the last sliding_window=128 tokens via DynamicSlidingWindowLayer / DeepseekV4HCACache.update.

self.keys = full[:, :, -self.sliding_window + 1 :, :]
        self.values = self.keys

The CSA compressor uses a Lightning Indexer that scores compressed KV blocks and keeps only the top index_topk=512 per query (index_n_heads=64, index_head_dim=128), used in compressed_sparse_attention layers of this config.

top_k_indices = index_scores.topk(top_k, dim=-1).indices  # [B, S, k]
            invalid = top_k_indices >= causal_threshold.unsqueeze(-1)
            return torch.where(invalid, torch.full_like(top_k_indices, -1), top_k_indices)

Each attention layer has a learnable per-head sink parameter concatenated into the softmax denominator before being dropped.

sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)

self.sinks = nn.Parameter(torch.empty(self.num_heads))

Queries are normalized with an unweighted RMSNorm per head after the q_b_proj, before rotary and the dot product; keys (the shared kv) are normalized with DeepseekV4RMSNorm as well.

self.q_b_norm = DeepseekV4UnweightedRMSNorm(eps=config.rms_norm_eps)

q = self.q_b_proj(q_residual).view(*hidden_shape).transpose(1, 2)
        q = self.q_b_norm(q)
        q = apply_rotary_pos_emb(q, cos, sin)

RoPE is applied only to the trailing qk_rope_head_dim=64 of the 512-dim head, leaving the leading nope channels untouched.

nope, rope = x[..., :-rope_dim], x[..., -rope_dim:]
    rotated = ((rope.float() * cos) + (rotate_half(rope).float() * sin)).to(x.dtype)
    return torch.cat([nope, rotated], dim=-1)

rope_scaling type is yarn and is applied to the 'compress' rope-type branch used by CSA/HCA compressor layers via the standard YaRN init function.

rope_init_fn = self.compute_default_rope_parameters
            if self.rope_type[layer_type] != "default":
                rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type[layer_type]]

Feed-forward layers route tokens to a subset of n_routed_experts=256 expert MLPs with num_experts_per_tok=6 chosen by a learned TopKRouter (or hash router for early layers).

indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)

Every MoE block runs a shared DeepseekV4MLP that every token passes through in addition to the routed experts, and n_shared_experts=1 in this config.

self.shared_experts = DeepseekV4MLP(config)

routed = self.experts(flat, indices, weights).view(batch, seq_len, hidden_dim)
        return routed + self.shared_experts(residual)

The router adds an e_score_correction_bias buffer to scores only for top-k selection (not for the weights used afterward), matching DeepSeek-V3's noaux_tc scheme, and topk_method is set to noaux_tc in this config.

indices = torch.topk(scores + self.e_score_correction_bias, self.top_k, dim=-1, sorted=False).indices
        weights = scores.gather(1, indices)

The config declares num_nextn_predict_layers=3 and the model explicitly ignores MTP weight keys on load, indicating the checkpoint carries MTP modules even though this file does not instantiate them.

_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]

Both the dense MLP and expert MLPs compute silu(gate(x)) * up(x) (with clamping), and hidden_act is silu in this config.

gate = self.gate_proj(x).clamp(max=self.limit)
        up = self.up_proj(x).clamp(min=-self.limit, max=self.limit)
        return self.down_proj(self.act_fn(gate) * up)