Open Models Almanac
ModelsHy4-preview

Tencent (Hunyuan) · usage rank #4

Hy4-preview

What Hy4-preview’s own documents say it is built from — every method with the sentence that describes it.

Parameters
770B / 49B active
Context
1M
Attention
sparse (indexer)Gated DeepSeek Sparse Attention (MLA) with IndexCache cross-layer index reuse
Experts
256 routed + 1 shared, top-8 (layer 1 dense)
Input
text
Licence
Apache 2.0
Released
2026-08-28

Rank by open-weight tokens on OpenRouter, week of 2026-10-26.

Against the consensus recipe

Its documents state 4 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 10 of the 19 architecture features checked — 8 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_hy_v4.py, configuration_hy_v4.py) and config.json at revision 705d81ee51, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention not in its codenot stated

The config overrides num_key_value_heads to equal num_attention_heads since MLA produces one KV per query head, so there is no grouping of…

show evidence

The config overrides num_key_value_heads to equal num_attention_heads since MLA produces one KV per query head, so there is no grouping of query heads onto fewer KV heads.

num_attention_heads = 64num_key_value_heads = 8
# MLA expands the latent to one key/value per query head, so keys are never grouped.
        self.num_key_value_heads = self.num_attention_heads
configuration_hy_v4.py · L140–141
Multi-head latent attention code only in its codenot stated

Keys and values are compressed into a shared low-rank KV latent (kv_lora_rank) and up-projected per head via kv_b_proj, with a decoupled…

show evidence

Keys and values are compressed into a shared low-rank KV latent (kv_lora_rank) and up-projected per head via kv_b_proj, with a decoupled RoPE key, DeepSeek-V2/V3 style MLA, and use_mla is true.

kv_lora_rank = 512q_lora_rank = 2048use_mla = true
self.kv_a_proj_with_mqa = nn.Linear(
            self.hidden_size,
            config.kv_lora_rank + config.qk_rope_head_dim,
            bias=config.attention_bias,
        )
        self.kv_a_layernorm = HYV4RMSNorm(config.kv_lora_rank)
        self.kv_b_proj = nn.Linear(
            config.kv_lora_rank,
            self.num_heads * (self.qk_nope_head_dim + self.v_head_dim),
            bias=False,
        )
modeling_hy_v4.py · L349–359
Sliding-window attention not in its codenot stated

No sliding_window key or windowed masking logic appears anywhere in the config or modeling code; layer_types are uniformly…

show evidence

No sliding_window key or windowed masking logic appears anywhere in the config or modeling code; layer_types are uniformly 'deepseek_sparse_attention'.

layer_types = ["deepseek_sparse_attention", "deepseek_sparse_attention", "deepseek_sparse_a…
Interleaved sliding-window and global attention not in its codenot stated

There is no interleaving of sliding-window and global layers; every layer_types entry is the same 'deepseek_sparse_attention' type.

show evidencelayer_types = ["deepseek_sparse_attention", "deepseek_sparse_attention", "deepseek_sparse_a…
Indexer-selected sparse attention (DeepSeek Sparse Attention) code only in its codenot stated

An HYV4Indexer module scores past tokens per head and each query attends only to the top index_topk selected tokens (DeepSeek Sparse…

show evidence

An HYV4Indexer module scores past tokens per head and each query attends only to the top index_topk selected tokens (DeepSeek Sparse Attention / DSA), enabled via use_dsa and layer_types set to deepseek_sparse_attention.

index_topk = 2048index_n_heads = 32use_dsa = true
topk = min(self.index_topk, index_scores.shape[-1])
        return index_scores.topk(topk, dim=-1).indices.to(torch.int32)  # [B, S, topk]
modeling_hy_v4.py · L257–258
self.skip_topk = config.indexer_types[layer_idx] == "shared"
        self.indexer = None if self.skip_topk else HYV4Indexer(config, layer_idx)
modeling_hy_v4.py · L369–370
Linear-attention or state-space layers alongside full attention not in its codenot stated

No linear-attention, Mamba, or Gated DeltaNet layer types or configuration keys exist; all layers are DeepSeek sparse (softmax) attention.

show evidencelayer_types = ["deepseek_sparse_attention", "deepseek_sparse_attention", "deepseek_sparse_a…
Gated DeltaNet layers not in its codenot stated

No gated delta-rule recurrent mixer code or related config keys (linear_num_key_heads, linear_conv_kernel_dim) are present.

Mamba-2 layers not in its codenot stated

No Mamba-2/SSD state-space block implementation or related config keys (mamba_num_heads, ssm_state_size) are present.

Learnable attention sink code only in its codenot stated

Each attention head has a learned sink parameter that is concatenated as an extra logit column before the softmax and then dropped from…

show evidence

Each attention head has a learned sink parameter that is concatenated as an extra logit column before the softmax and then dropped from the output probabilities, enabled by learnable_sink.

learnable_sink = truelearnable_sink_init = 0.0
sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_hy_v4.py · L289–290
self.sinks = nn.Parameter(torch.full((self.num_heads,), config.learnable_sink_init))
modeling_hy_v4.py · L373
Gated attention output code only in its codenot stated

An input-dependent gate projection produces per-head/per-element values passed through sigmoid and multiplied into the attention output…

show evidence

An input-dependent gate projection produces per-head/per-element values passed through sigmoid and multiplied into the attention output before the output projection, consistent with gated_mla=true.

gated_mla = truegating_type = "elementwise"
gate_states = self.gate_proj(hidden_states).view(batch_size, seq_length, -1, self.gate_projection_size)
modeling_hy_v4.py · L408
attn_output = attn_output * torch.sigmoid(gate_states)
modeling_hy_v4.py · L478
QK normalization not in its codenot stated

Main attention queries/keys (q_b_proj, kv_b_proj outputs) are not normalized before the dot product; only the indexer's single key head…

show the whole verdict

Main attention queries/keys (q_b_proj, kv_b_proj outputs) are not normalized before the dot product; only the indexer's single key head uses a LayerNorm (k_norm), which is part of the sparse indexer, not the main QK path, and there is no use_qk_norm style config switch.

Partial RoPE code only in its codenot stated

RoPE is applied only to the qk_rope_head_dim slice of each query/key (64 of the 256 total qk_head_dim), the qk_nope_head_dim portion…

show evidence

RoPE is applied only to the qk_rope_head_dim slice of each query/key (64 of the 256 total qk_head_dim), the qk_nope_head_dim portion carries no rotary signal, matching MLA's decoupled RoPE key.

qk_rope_head_dim = 64qk_nope_head_dim = 192
q_pass, q_rot = torch.split(q_states, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
modeling_hy_v4.py · L412
q_rot, k_rot = apply_rotary_pos_emb(q_rot, k_rot, cos, sin)
modeling_hy_v4.py · L422
YaRN RoPE scaling not in its codenot stated

rope_parameters.rope_type is set to 'default' rather than 'yarn', so although yarn-related mscale helper code exists it is not activated…

show evidence

rope_parameters.rope_type is set to 'default' rather than 'yarn', so although yarn-related mscale helper code exists it is not activated by this config.

rope_parameters = {"rope_theta": 10000000, "rope_type": "default"}
Layers without positional encoding (NoPE) not in its codenot stated

Every attention layer applies rotary embeddings to its rope slice uniformly (via the shared HYV4RotaryEmbedding and apply_rotary_pos_emb…

show the whole verdict

Every attention layer applies rotary embeddings to its rope slice uniformly (via the shared HYV4RotaryEmbedding and apply_rotary_pos_emb call in every HYV4Attention forward); there is no config or code path that skips RoPE entirely for a subset of softmax attention layers.

Mixture of experts in its codeadopted

Most decoder layers use HYV4MoE with a learned HYV4TopkRouter that routes each token to num_experts_per_tok of n_routed_experts experts…

show evidence

Most decoder layers use HYV4MoE with a learned HYV4TopkRouter that routes each token to num_experts_per_tok of n_routed_experts experts, per mlp_layer_types='sparse'.

n_routed_experts = 256num_experts_per_tok = 8
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_hy_v4.py · L531–532
self.mlp = HYV4MoE(config) if config.mlp_layer_types[layer_idx] == "sparse" else HYV4MLP(config)
modeling_hy_v4.py · L673
Shared expert in its codeadopted

HYV4MoE instantiates a shared_experts HYV4MLP sized by n_shared_experts (1) that is added to every token's output alongside the routed…

show evidence

HYV4MoE instantiates a shared_experts HYV4MLP sized by n_shared_experts (1) that is added to every token's output alongside the routed experts.

n_shared_experts = 1
self.shared_experts = HYV4MLP(
            config=config, intermediate_size=config.moe_intermediate_size * config.n_shared_experts
        )
modeling_hy_v4.py · L588–590
hidden_states = hidden_states + self.shared_experts(residuals)
modeling_hy_v4.py · L598
Auxiliary-loss-free load balancing (selection bias) code only in its codenot stated

The router adds e_score_correction_bias to the sigmoid scores only for top-k expert selection, while unbiased scores are used for the…

show evidence

The router adds e_score_correction_bias to the sigmoid scores only for top-k expert selection, while unbiased scores are used for the output weights, DeepSeek-V3 style.

scores = router_logits.sigmoid()
        scores_for_choice = scores + self.e_score_correction_bias
modeling_hy_v4.py · L515–516
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
        topk_weights = scores.gather(1, topk_indices)
modeling_hy_v4.py · L531–532
Multi-token prediction layers code only in its codenot stated

num_nextn_predict_layers is set to 1 in config, and the modeling code explicitly ignores mtp-layer weights on load, confirming declared…

show evidence

num_nextn_predict_layers is set to 1 in config, and the modeling code explicitly ignores mtp-layer weights on load, confirming declared extra prediction modules exist in the checkpoint.

num_nextn_predict_layers = 1mtp_loss_factor = 0.1
_keys_to_ignore_on_load_unexpected = [r"model\.mtp_layers\..*"]
modeling_hy_v4.py · L750
SwiGLU feed-forward code only in its codenot stated

Both the dense HYV4MLP and the routed HYV4Experts compute SiLU(gate(x)) * up(x), i.e. SwiGLU, matching hidden_act='silu'.

show evidencehidden_act = "silu"
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
modeling_hy_v4.py · L495
return F.silu(gate) * up
modeling_hy_v4.py · L575

token mixer 2

sparse attention 2

Gated DeepSeek Sparse Attention core

“the attention module employs Gated DeepSeek Sparse Attention (Gated DSA) with IndexCache for cross-layer sparse index reuse”

sparse attention indexer 1

IndexCache core

“with IndexCache for cross-layer sparse index reuse”

channel mixer 3

mixture of experts 3

Mixture of experts core

“The backbone consists of 78 layers, where the first layer uses a standard dense FFN and the remaining 77 layers replace it with MoE”

expert routing 1

Top-k expert routing core

“every token activates the top-8 routed experts along with the shared expert”

shared experts 1

Shared expert core

“every token activates the top-8 routed experts along with the shared expert”

normalization & residual 1

Identity Hyper-Connections core

“The residual pathway uses iHC (identity Hyper-Connections) to expand inter-layer information flow.”

prediction head 1

MTP speculative decoding core

“1 native MTP layer (10B total parameters, 0.7B activated) is built in for speculative decoding”

filed at the root 1

Product co-design for training data used

“deep co-design with products such as WorkBuddy”

data sourcing 1

Expert co-created training data used

“developed using high-quality training data co-created with Tencent experts across software engineering, gaming, finance, security, and…”

synthetic data 1

Training-data construction around expert work products used

“we partnered with top experts inside Tencent ... and built training data around the work they ship”

filed at the root 1

Communication optimization used

“optimization on areas such as operator fusion and communication optimization”

decoding strategy 3

Multi-Token Prediction used

“1 native MTP layer (10B total parameters, 0.7B activated) is built in for speculative decoding.”

Speculative decoding used

“is built in for speculative decoding”

NEXTN speculative decoding used

“--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4”

reasoning control 2

Chain-of-thought reasoning default

“Reasoning mode: Defaults to "high" (deep chain-of-thought), which suits complex tasks such as math, coding, and reasoning.”

Thinking mode used

“So it looks like there are just two reasoning effort levels: "high" (the default) and "no_think" (reason by disabled).”

inference quantization 2

FP8 used

“Hy4 preview-FP8 model weights”

Large-model quantization and speculative sampling optional

“AngelSlim supports a comprehensive suite of compression tools for large-scale multimodal models, including common quantization algorithms…”

serving parallelism 1

Tensor parallelism used

“--tensor-parallel-size 8”

inference kernel 1

Kernel fusion used

“optimization on areas such as operator fusion and communication optimization”

agentic scaffolding 2

Automatic tool choice used

“--enable-auto-tool-choice”

hy_v4 tool-call parser used

“--tool-call-parser hy_v4”

inference engine 1

vLLM or SGLang serving used

“For production serving, we recommend using vLLM or SGLang.”

kernel & quantization library 2

AngelSlim used

“We provide AngelSlim, a more accessible, comprehensive, and efficient toolkit for large model compression. AngelSlim supports a…”

FLASHMLA_SPARSE used

“--attention-backend FLASHMLA_SPARSE”

human & real-world evaluation 1

Blind expert evaluation of engineering tasks used

“we ran a blind side-by-side evaluation: 163 internal experts rated model outputs on 203 engineering tasks.”

filed at the root 1

Early release with feedback-driven iteration used

“we would rather ship early and hear what breaks — that's what made Hy3 substantially better”

unfiled 8

automated optimization of training and data strategies used

“participating for the first time in the automated optimization of training methods, data strategies, evaluation frameworks, and low-level…”

autonomous inference system optimization used

“autonomously analyzed bottlenecks in its inference system and carried out multiple rounds of optimization on areas such as operator fusion…”

chat templates for model prompting used

“I recently started using model chat templates to better understand their capabilities.”

Co-designing the model with downstream products used

“We also continue to co-design Hy4 preview with Tencent products like CodeBuddy and WorkBuddy”

pre-training and post-training used

“the advances in both pre-training and post-training have led to a major leap in overall intelligence”

preview-first release approach used

“Through a preview-first approach, followed by official releases, Hunyuan continuously incorporates real-world feedback into its research…”

Reasoning parser hy_v4 used

“--reasoning-parser hy_v4”

recursive self-improvement loop used

“This established an early-stage recursive self-improvement loop.”

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

SourceKindPublisherStatus
tencent.com/tencent-releases-and-open-sources-tencent-hy4-previewofficial blogTencentread
hy.tencent.com/research/hy4-previewofficial blogTencent Hynot fetched
huggingface.co/tencent/Hy4-previewmodel cardTencent Hunyuanread
github.com/Tencent-Hunyuan/Hy4-previewcode repoTencent Hunyuanread
recipes.vllm.ai/tencent/Hy4-previewvendor docsvLLMread
openrouter.ai/tencent/hy4-previewvendor docsOpenRouterread
technode.com/2026/08/28/tencent-open-sources-hy4-preview-with-770b-para…newsTechNoderead
simonwillison.net/2026/Aug/29/hy4third party analysisSimon Willisonread
↑