Model techniques map
ModelsHy4-preview

Tencent (Hunyuan) · usage rank #3 · released 2026-08-28

Hy4-preview

What Hy4-preview’s own documents say it is built from — every method with the sentence that describes it.

OpenRouterHugging Face

Curator’s note770B total / 49B active MoE: 78 layers (layer 1 dense FFN, the other 77 MoE with 256 routed + 1 shared expert, top-8 per token) plus a native 10B/0.7B MTP layer for speculative decoding. Attention is Gated DeepSeek Sparse Attention with IndexCache cross-layer index reuse; the residual pathway uses identity Hyper-Connections (4 residual streams). 1M context, Apache 2.0. No Tencent system report for Hy4: the model card is the architecture document, and the two arXiv ids it cites (2512.02556 DSA, 2603.12201 IndexCache) are the cited methods' own papers, not Hy4's — they are deliberately not listed as Hy4 sources.

Rank by open-weight tokens on OpenRouter, week of 2026-09-21.

Against the consensus recipe

Its documents state 4 of the 15 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something the documents do not say, not something the model lacks.

  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 9 of the 19 architecture features checked — 7 of them are not stated in its documents.

Documents say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_hy_v4.py, configuration_hy_v4.py) and config.json at revision 705d81ee51, by anthropic/claude-sonnet-5 on 2026-09-25.

FeatureCodeIts documentsEvidence
Grouped-query attention not in its codenot stated

The config sets num_key_value_heads equal to num_attention_heads (MLA overrides it in __post_init__), so there is no head grouping; attention uses MLA instead of GQA.

num_attention_heads = 64num_key_value_heads = 8
# MLA expands the latent to one key/value per query head, so keys are never grouped.
        self.num_key_value_heads = self.num_attention_heads
configuration_hy_v4.py · L138–139
Multi-head latent attention code only in its codenot stated

Attention down-projects hidden states into a shared kv_lora_rank latent (kv_a_proj_with_mqa/kv_a_layernorm) and up-projects per head via kv_b_proj, DeepSeek-V2/V3 style MLA with a decoupled RoPE key.

kv_lora_rank = 512q_lora_rank = 2048qk_rope_head_dim = 64
self.kv_a_proj_with_mqa = nn.Linear(
            self.hidden_size,
            config.kv_lora_rank + config.qk_rope_head_dim,
            bias=config.attention_bias,
        )
        self.kv_a_layernorm = HYV4RMSNorm(config.kv_lora_rank)
modeling_hy_v4.py · L361–366
Sliding-window attention not in its codenot stated

No sliding_window config key or windowed masking code is present; all layers use the DeepSeek sparse (indexer-selected) attention over the full causal context.

Interleaved sliding-window and global attention not in its codenot stated

layer_types lists only "deepseek_sparse_attention" for every layer, with no sliding-window layer type interleaved.

layer_types = ["deepseek_sparse_attention", "deepseek_sparse_attention", "deepseek_sparse_a…
Indexer-selected sparse attention (DeepSeek Sparse Attention) code only in its codenot stated

HYV4Indexer scores past tokens with a lightweight head and each query attends only to the index_topk selected tokens via a scattered sparse mask.

index_topk = 2048index_n_heads = 32index_head_dim = 128
topk = min(self.index_topk, index_scores.shape[-1])
        return index_scores.topk(topk, dim=-1).indices.to(torch.int32)  # [B, S, topk]
modeling_hy_v4.py · L268–269
index_mask = (
                topk_indices.new_ones((batch_size, seq_length, key_states.shape[2]), dtype=torch.bool)
                .scatter(-1, topk_indices.long(), False)
                .unsqueeze(1)
            )
modeling_hy_v4.py · L461–465
Linear-attention or state-space layers alongside full attention not in its codenot stated

All layers use the same DeepSeek sparse attention mechanism; there is no linear-attention or SSM layer type in layer_types.

layer_types = ["deepseek_sparse_attention", "deepseek_sparse_attention", "deepseek_sparse_a…
Gated DeltaNet layers not in its codenot stated

No gated delta-rule recurrent mixer code or config keys (linear_num_key_heads, linear_conv_kernel_dim) exist in this model.

Mamba-2 layers not in its codenot stated

No Mamba-2/SSD state-space code or config keys (mamba_num_heads, ssm_state_size) are present.

Learnable attention sink code only in its codenot stated

Each attention head has a learned sink logit parameter concatenated to the attention logits before the softmax denominator.

learnable_sink = truelearnable_sink_init = 0.0
sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_hy_v4.py · L300–301
self.sinks = nn.Parameter(torch.full((self.num_heads,), config.learnable_sink_init))
modeling_hy_v4.py · L385
Gated attention output code only in its codenot stated

The attention output is multiplied elementwise by a sigmoid gate computed from a separate gate_proj before the output projection.

gated_mla = truegating_type = "elementwise"
attn_output = attn_output * torch.sigmoid(gate_states)
modeling_hy_v4.py · L490
self.gate_proj = nn.Linear(config.hidden_size, self.num_heads * self.gate_projection_size, bias=False)
modeling_hy_v4.py · L384
QK normalization not in its codenot stated

There is no RMSNorm/LayerNorm applied to the main query/key projections of HYV4Attention (only the DSA indexer's separate k_norm is normalized), so QK-norm on the main attention path is absent.

Partial RoPE code only in its codenot stated

RoPE is applied only to the qk_rope_head_dim slice (64 of 256) of each query/key head, with the remaining qk_nope_head_dim portion carrying no position signal, MLA decoupled-RoPE style.

qk_rope_head_dim = 64qk_nope_head_dim = 192
q_pass, q_rot = torch.split(q_states, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
modeling_hy_v4.py · L424
YaRN RoPE scaling not in its codenot stated

rope_parameters.rope_type is "default" rather than "yarn", so the YaRN scaling code path (yarn_apply_mscale checking rope_type != default) is not triggered.

rope_parameters = {"rope_theta": 10000000, "rope_type": "default"}
def yarn_apply_mscale(rope_parameters, scaling):
    if rope_parameters.get("rope_type", "default") != "default":
modeling_hy_v4.py · L321–322
Layers without positional encoding (NoPE) not in its codenot stated

layer_types shows every layer is the same deepseek_sparse_attention type and all apply RoPE to their rope slice; there is no subset of full-attention layers that skip positional encoding entirely.

layer_types = ["deepseek_sparse_attention", "deepseek_sparse_attention", "deepseek_sparse_a…
Mixture of experts in its code core

HYV4TopkRouter routes each token to num_experts_per_tok of n_routed_experts expert MLPs via sigmoid-scored top-k selection, used in every layer where mlp_layer_types is "sparse".

n_routed_experts = 256num_experts_per_tok = 8
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_hy_v4.py · L543–544
Shared expert in its code core

HYV4MoE adds a shared_experts HYV4MLP with intermediate size scaled by n_shared_experts=1, applied to every token in addition to the routed experts.

n_shared_experts = 1
self.shared_experts = HYV4MLP(
            config=config, intermediate_size=config.moe_intermediate_size * config.n_shared_experts
        )
modeling_hy_v4.py · L600–602
hidden_states = hidden_states + self.shared_experts(residuals)
modeling_hy_v4.py · L610
Auxiliary-loss-free load balancing (selection bias) code only in its codenot stated

The router adds an e_score_correction_bias buffer to the sigmoid scores solely for top-k expert selection, while the actual output weighting uses the unbiased scores gathered afterward, DeepSeek-V3 style.

scores_for_choice = scores + self.e_score_correction_bias
modeling_hy_v4.py · L528
topk_weights = scores.gather(1, topk_indices)
modeling_hy_v4.py · L544
Multi-token prediction layers code unclear core

The config declares num_nextn_predict_layers=1 and mtp_loss_factor, and the modeling file references model.mtp_layers.* weights to ignore on load, but the MTP module itself is not defined/built in the given files.

num_nextn_predict_layers = 1mtp_loss_factor = 0.1
_keys_to_ignore_on_load_unexpected = [r"model\.mtp_layers\..*"]
modeling_hy_v4.py · L761
SwiGLU feed-forward code only in its codenot stated

Both the dense HYV4MLP and the MoE HYV4Experts compute silu(gate(x)) * up(x), a SwiGLU feed-forward, with hidden_act set to silu.

hidden_act = "silu"
def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj
modeling_hy_v4.py · L506–508
return F.silu(gate) * up
modeling_hy_v4.py · L587

model architecture 9

token mixer 2

sparse attention 2

Gated DeepSeek Sparse Attention core

sparse attention indexer 1

IndexCache core

channel mixer 5

normalization & residual 1

Identity Hyper-Connections core

prediction head 1

Multi-Token Prediction core

data curation 3

filed at the root 1

Product co-design for training data used

data sourcing 1

Expert co-created training data used

synthetic data 1

Training-data construction around expert work products used

inference & serving 12

filed at the root 1

communication optimization used

decoding strategy 3

Speculative decoding used
Multi-Token Prediction used
NEXTN speculative decoding used

reasoning control 2

Chain-of-thought reasoning default
Configurable reasoning effort used

inference quantization 2

FP8 used
Large-model compression using quantization and speculative sampling optional

serving parallelism 1

Tensor parallelism (degree 8) used

inference kernel 1

Kernel fusion used

agentic scaffolding 2

Automatic tool choice used
Hy v4 tool-call parser used

software implementation 3

inference engine 1

vLLM or SGLang serving used

kernel & quantization library 2

AngelSlim used
FLASHMLA_SPARSE used

evaluation 1

human & real-world evaluation 1

Blind side-by-side expert evaluation used

other 1

filed at the root 1

Early Release with Feedback-Driven Iteration used

unfiled 8

automated optimization of training and data strategies used
autonomous inference system optimization used
chat templates for model prompting used
Co-designing the model with downstream products used
pre-training and post-training used
preview-first release approach used
Reasoning parser hy_v4 used
recursive self-improvement loop used

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated document list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

DocumentKindPublisherStatus
tencent.com/tencent-releases-and-open-sources-tencent-hy4-previewofficial blogTencentread
hy.tencent.com/research/hy4-previewofficial blogTencent Hynot fetched
huggingface.co/tencent/Hy4-previewmodel cardTencent Hunyuanread
github.com/Tencent-Hunyuan/Hy4-previewcode repoTencent Hunyuanread
recipes.vllm.ai/tencent/Hy4-previewvendor docsvLLMread
openrouter.ai/tencent/hy4-previewvendor docsOpenRouterread
technode.com/2026/08/28/tencent-open-sources-hy4-preview-with-770b-para…newsTechNoderead
simonwillison.net/2026/Aug/29/hy4third party analysisSimon Willisonread