Tencent (Hunyuan) · usage rank #4
Hy4-preview
What Hy4-preview’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 770B / 49B active
- Context
- 1M
- Attention
- sparse (indexer)Gated DeepSeek Sparse Attention (MLA) with IndexCache cross-layer index reuse
- Experts
- 256 routed + 1 shared, top-8 (layer 1 dense)
- Input
- text
- Licence
- Apache 2.0
- Released
- 2026-08-28
Rank by open-weight tokens on OpenRouter, week of 2026-10-26.
Against the consensus recipe
Its documents state 4 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictionnot in its documents
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modeused
- Multi-Token Predictionused
- FP8used
Checked in its code
Its code shows 10 of the 19 architecture features checked — 8 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_hy_v4.py, configuration_hy_v4.py) and config.json at revision 705d81ee51, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention | not in its code | not stated | The config overrides num_key_value_heads to equal num_attention_heads since MLA produces one KV per query head, so there is no grouping of… show evidenceThe config overrides num_key_value_heads to equal num_attention_heads since MLA produces one KV per query head, so there is no grouping of query heads onto fewer KV heads. num_attention_heads = 64num_key_value_heads = 8 |
| Multi-head latent attention code only | in its code | not stated | Keys and values are compressed into a shared low-rank KV latent (kv_lora_rank) and up-projected per head via kv_b_proj, with a decoupled… show evidenceKeys and values are compressed into a shared low-rank KV latent (kv_lora_rank) and up-projected per head via kv_b_proj, with a decoupled RoPE key, DeepSeek-V2/V3 style MLA, and use_mla is true. kv_lora_rank = 512q_lora_rank = 2048use_mla = true |
| Sliding-window attention | not in its code | not stated | No sliding_window key or windowed masking logic appears anywhere in the config or modeling code; layer_types are uniformly… show evidenceNo sliding_window key or windowed masking logic appears anywhere in the config or modeling code; layer_types are uniformly 'deepseek_sparse_attention'. layer_types = ["deepseek_sparse_attention", "deepseek_sparse_attention", "deepseek_sparse_a… |
| Interleaved sliding-window and global attention | not in its code | not stated | There is no interleaving of sliding-window and global layers; every layer_types entry is the same 'deepseek_sparse_attention' type. |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) code only | in its code | not stated | An HYV4Indexer module scores past tokens per head and each query attends only to the top index_topk selected tokens (DeepSeek Sparse… show evidenceAn HYV4Indexer module scores past tokens per head and each query attends only to the top index_topk selected tokens (DeepSeek Sparse Attention / DSA), enabled via use_dsa and layer_types set to deepseek_sparse_attention. index_topk = 2048index_n_heads = 32use_dsa = true |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | No linear-attention, Mamba, or Gated DeltaNet layer types or configuration keys exist; all layers are DeepSeek sparse (softmax) attention. |
| Gated DeltaNet layers | not in its code | not stated | No gated delta-rule recurrent mixer code or related config keys (linear_num_key_heads, linear_conv_kernel_dim) are present. |
| Mamba-2 layers | not in its code | not stated | No Mamba-2/SSD state-space block implementation or related config keys (mamba_num_heads, ssm_state_size) are present. |
| Learnable attention sink code only | in its code | not stated | Each attention head has a learned sink parameter that is concatenated as an extra logit column before the softmax and then dropped from… show evidenceEach attention head has a learned sink parameter that is concatenated as an extra logit column before the softmax and then dropped from the output probabilities, enabled by learnable_sink. learnable_sink = truelearnable_sink_init = 0.0 |
| Gated attention output code only | in its code | not stated | An input-dependent gate projection produces per-head/per-element values passed through sigmoid and multiplied into the attention output… show evidenceAn input-dependent gate projection produces per-head/per-element values passed through sigmoid and multiplied into the attention output before the output projection, consistent with gated_mla=true. gated_mla = truegating_type = "elementwise" |
| QK normalization | not in its code | not stated | Main attention queries/keys (q_b_proj, kv_b_proj outputs) are not normalized before the dot product; only the indexer's single key head… show the whole verdictMain attention queries/keys (q_b_proj, kv_b_proj outputs) are not normalized before the dot product; only the indexer's single key head uses a LayerNorm (k_norm), which is part of the sparse indexer, not the main QK path, and there is no use_qk_norm style config switch. |
| Partial RoPE code only | in its code | not stated | RoPE is applied only to the qk_rope_head_dim slice of each query/key (64 of the 256 total qk_head_dim), the qk_nope_head_dim portion… show evidenceRoPE is applied only to the qk_rope_head_dim slice of each query/key (64 of the 256 total qk_head_dim), the qk_nope_head_dim portion carries no rotary signal, matching MLA's decoupled RoPE key. qk_rope_head_dim = 64qk_nope_head_dim = 192 |
| YaRN RoPE scaling | not in its code | not stated | rope_parameters.rope_type is set to 'default' rather than 'yarn', so although yarn-related mscale helper code exists it is not activated… show evidencerope_parameters.rope_type is set to 'default' rather than 'yarn', so although yarn-related mscale helper code exists it is not activated by this config. rope_parameters = {"rope_theta": 10000000, "rope_type": "default"} |
| Layers without positional encoding (NoPE) | not in its code | not stated | Every attention layer applies rotary embeddings to its rope slice uniformly (via the shared HYV4RotaryEmbedding and apply_rotary_pos_emb… show the whole verdictEvery attention layer applies rotary embeddings to its rope slice uniformly (via the shared HYV4RotaryEmbedding and apply_rotary_pos_emb call in every HYV4Attention forward); there is no config or code path that skips RoPE entirely for a subset of softmax attention layers. |
| Mixture of experts | in its code | adopted | Most decoder layers use HYV4MoE with a learned HYV4TopkRouter that routes each token to num_experts_per_tok of n_routed_experts experts… show evidenceMost decoder layers use HYV4MoE with a learned HYV4TopkRouter that routes each token to num_experts_per_tok of n_routed_experts experts, per mlp_layer_types='sparse'. n_routed_experts = 256num_experts_per_tok = 8 |
| Shared expert | in its code | adopted | HYV4MoE instantiates a shared_experts HYV4MLP sized by n_shared_experts (1) that is added to every token's output alongside the routed… show evidenceHYV4MoE instantiates a shared_experts HYV4MLP sized by n_shared_experts (1) that is added to every token's output alongside the routed experts. n_shared_experts = 1 |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | The router adds e_score_correction_bias to the sigmoid scores only for top-k expert selection, while unbiased scores are used for the… show evidenceThe router adds e_score_correction_bias to the sigmoid scores only for top-k expert selection, while unbiased scores are used for the output weights, DeepSeek-V3 style. |
| Multi-token prediction layers code only | in its code | not stated | num_nextn_predict_layers is set to 1 in config, and the modeling code explicitly ignores mtp-layer weights on load, confirming declared… show evidencenum_nextn_predict_layers is set to 1 in config, and the modeling code explicitly ignores mtp-layer weights on load, confirming declared extra prediction modules exist in the checkpoint. num_nextn_predict_layers = 1mtp_loss_factor = 0.1 |
| SwiGLU feed-forward code only | in its code | not stated | Both the dense HYV4MLP and the routed HYV4Experts compute SiLU(gate(x)) * up(x), i.e. SwiGLU, matching hidden_act='silu'. show evidencehidden_act = "silu" |
token mixer 2
sparse attention 2
“the attention module employs Gated DeepSeek Sparse Attention (Gated DSA) with IndexCache for cross-layer sparse index reuse”
sparse attention indexer 1
“with IndexCache for cross-layer sparse index reuse”
channel mixer 3
mixture of experts 3
“The backbone consists of 78 layers, where the first layer uses a standard dense FFN and the remaining 77 layers replace it with MoE”
expert routing 1
“every token activates the top-8 routed experts along with the shared expert”
normalization & residual 1
“The residual pathway uses iHC (identity Hyper-Connections) to expand inter-layer information flow.”
prediction head 1
“1 native MTP layer (10B total parameters, 0.7B activated) is built in for speculative decoding”
filed at the root 1
“deep co-design with products such as WorkBuddy”
data sourcing 1
“developed using high-quality training data co-created with Tencent experts across software engineering, gaming, finance, security, and…”
synthetic data 1
“we partnered with top experts inside Tencent ... and built training data around the work they ship”
filed at the root 1
“optimization on areas such as operator fusion and communication optimization”
decoding strategy 3
“1 native MTP layer (10B total parameters, 0.7B activated) is built in for speculative decoding.”
“is built in for speculative decoding”
“--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4”
reasoning control 2
“Reasoning mode: Defaults to "high" (deep chain-of-thought), which suits complex tasks such as math, coding, and reasoning.”
“So it looks like there are just two reasoning effort levels: "high" (the default) and "no_think" (reason by disabled).”
inference quantization 2
“Hy4 preview-FP8 model weights”
“AngelSlim supports a comprehensive suite of compression tools for large-scale multimodal models, including common quantization algorithms…”
serving parallelism 1
“--tensor-parallel-size 8”
inference kernel 1
“optimization on areas such as operator fusion and communication optimization”
agentic scaffolding 2
“--enable-auto-tool-choice”
“--tool-call-parser hy_v4”
inference engine 1
“For production serving, we recommend using vLLM or SGLang.”
kernel & quantization library 2
“We provide AngelSlim, a more accessible, comprehensive, and efficient toolkit for large model compression. AngelSlim supports a…”
“--attention-backend FLASHMLA_SPARSE”
human & real-world evaluation 1
“we ran a blind side-by-side evaluation: 163 internal experts rated model outputs on 203 engineering tasks.”
other 1
filed at the root 1
“we would rather ship early and hear what breaks — that's what made Hy3 substantially better”
unfiled 8
“participating for the first time in the automated optimization of training methods, data strategies, evaluation frameworks, and low-level…”
“autonomously analyzed bottlenecks in its inference system and carried out multiple rounds of optimization on areas such as operator fusion…”
“I recently started using model chat templates to better understand their capabilities.”
“We also continue to co-design Hy4 preview with Tencent products like CodeBuddy and WorkBuddy”
“the advances in both pre-training and post-training have led to a major leap in overall intelligence”
“Through a preview-first approach, followed by official releases, Hunyuan continuously incorporates real-world feedback into its research…”
“--reasoning-parser hy_v4”
“This established an early-stage recursive self-improvement loop.”
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Hy4-preview architecture card — LLM Architecture Gallery explainer sebastianraschka.com
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| tencent.com/tencent-releases-and-open-sources-tencent-hy4-preview | official blog | Tencent | read |
| hy.tencent.com/research/hy4-preview | official blog | Tencent Hy | not fetched |
| huggingface.co/tencent/Hy4-preview | model card | Tencent Hunyuan | read |
| github.com/Tencent-Hunyuan/Hy4-preview | code repo | Tencent Hunyuan | read |
| recipes.vllm.ai/tencent/Hy4-preview | vendor docs | vLLM | read |
| openrouter.ai/tencent/hy4-preview | vendor docs | OpenRouter | read |
| technode.com/2026/08/28/tencent-open-sources-hy4-preview-with-770b-para… | news | TechNode | read |
| simonwillison.net/2026/Aug/29/hy4 | third party analysis | Simon Willison | read |