Tencent (Hunyuan) · usage rank #9
Hy3
What Hy3’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 295B / 21B active
- Context
- 256K
- Attention
- full (GQA/MHA)GQA with QK-norm, every layer
- Experts
- 192 routed + 1 shared, top-8
- Input
- text
- Licence
- Apache 2.0
- Released
- 2026-07-06
Rank by open-weight tokens on OpenRouter, week of 2026-10-26.
Against the consensus recipe
Its documents state 6 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningused
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modedefault
- Multi-Token Predictionnot in its documents
- FP8used
Checked in its code
Its code shows 6 of the 19 architecture features checked — 3 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_hy_v3.py, configuration_hy_v3.py) and config.json at revision a960ebc3da, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention | in its code | adopted | 64 query heads share 8 KV heads via repeat_kv in every attention layer. show evidencenum_attention_heads = 64num_key_value_heads = 8 |
| Multi-head latent attention | not in its code | not stated | Attention uses standard separate q/k/v projections sized by head_dim with no low-rank latent kv projection anywhere in the code. |
| Sliding-window attention | not in its code | not stated | The model builder explicitly uses a full causal mask with no sliding window mechanism. show evidence |
| Interleaved sliding-window and global attention | not in its code | not stated | There is no layer_types or sliding_window_pattern config, and all attention layers use the same full causal mask. |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | not in its code | not stated | No indexer module or top-k token selection mechanism exists in the attention code, and there are no index_topk-related config keys. |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | All decoder layers use the same HYV3Attention softmax attention; there is no recurrent/linear mixer layer type. show evidence |
| Gated DeltaNet layers | not in its code | not stated | No delta-rule recurrent state or decay gate implementation is present in the code. |
| Mamba-2 layers | not in its code | not stated | No SSM/Mamba-2 selective state-space block or related config keys (mamba_num_heads, ssm_state_size) appear in this model. |
| Learnable attention sink | not in its code | not stated | The attention forward computes a standard softmax over query-key scores with no extra learned sink logit column. show evidence |
| Gated attention output | not in its code | not stated | The attention output is passed directly to o_proj with no sigmoid gating multiplication before the output projection. show evidence |
| QK normalization code only | in its code | not stated | Queries and keys are each passed through a per-head RMSNorm before RoPE and the attention dot product, and qk_norm is enabled in config. show evidenceqk_norm = true |
| Partial RoPE | not in its code | not stated | RoPE is computed over the full head_dim (dim = head_dim) and applied to the whole query/key vectors, with no partial_rotary_factor or… show evidenceRoPE is computed over the full head_dim (dim = head_dim) and applied to the whole query/key vectors, with no partial_rotary_factor or rope_dim splitting. |
| YaRN RoPE scaling | not in its code | not stated | rope_parameters.rope_type is set to 'default', not yarn, so no YaRN scaling is applied. |
| Layers without positional encoding (NoPE) | not in its code | not stated | RoPE (via position_embeddings) is applied uniformly to every attention layer with no per-layer no-rope switch. show evidence |
| Mixture of experts | in its code | adopted | Feed-forward layers route tokens to 8 of 192 experts via a learned sigmoid router (HYV3TopKRouter) in all but the first dense layer. show evidencenum_experts = 192num_experts_per_tok = 8 |
| Shared expert | in its code | adopted | Each MoE layer includes a shared HYV3MLP expert sized by num_shared_experts=1 that every token passes through and adds to the routed output. show evidencenum_shared_experts = 1 |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | The router adds e_score_correction_bias to the sigmoid routing scores only for top-k selection, not for weighting, matching… show evidenceThe router adds e_score_correction_bias to the sigmoid routing scores only for top-k selection, not for weighting, matching DeepSeek-V3-style bias balancing. moe_router_enable_expert_bias = true |
| Multi-token prediction layers documents disagree | not in its code | adopted | Config declares num_nextn_predict_layers=1, but the model explicitly ignores/skips those weights and builds no MTP module. show evidencenum_nextn_predict_layers = 1 |
| SwiGLU feed-forward code only | in its code | not stated | Both the dense MLP and the MoE experts compute silu(gate(x)) * up(x) before the down projection. show evidencehidden_act = "silu" |
token mixer 1
softmax attention 1
grouped-query attention 1
“Attention Heads | 64 (GQA, 8 KV heads, head dim 128)”
channel mixer 4
mixture of experts 4
“Hy3 is a hybrid fast-and-slow-thinking model built on a Mixture-of-Experts (MoE) architecture”
“192 routed experts (top-8) + 1 shared expert”
expert routing 1
“Number of Experts | 192 experts, top-8 activated”
prediction head 1
“Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model with 21B active parameters and 3.8B MTP layer parameters”
training stability 1
“The anti-hallucination training reduces fabricated clauses and misquotes.”
data filtering 2
“we implemented fine-grained data cleaning and training constraints”
“Hy3 implements data cleaning and training constraints based on the principle of 'answering only when there is evidence, and clearly…”
filed at the root 1
“Joint SFT and RL improved coreference and constraint tracking.”
supervised fine-tuning 2
“Through joint optimization of SFT and RL”
“Hy3 provides a complete model finetuning pipeline”
reinforcement learning algorithm 2
“through strengthened reinforcement learning and enhanced data quality and diversity”
“Hy3 supports GRPO reinforcement learning training with verl”
decoding strategy 3
“for speculative decoding”
“--speculative-algorithm EAGLE”
“AngelSlim supports a comprehensive suite of compression tools ... including ... speculative sampling”
reasoning control 3
“reasoning_effort: "no_think" (default, direct response), "low", "high" (deep chain-of-thought)”
“--reasoning-parser hy_v3”
“reasoning_effort: 'no_think' (default, direct response), 'low', 'high' (deep chain-of-thought)”
inference quantization 3
“Hy3-FP8 | FP8 quantized instruct model”
“--kv-cache-dtype fp8_e4m3”
“AngelSlim supports a comprehensive suite of compression tools ... including common quantization algorithms, low-bit quantization”
serving parallelism 1
“export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm”
inference kernel 3
“fall back to the Triton MoE kernel”
“Enable the attention backend with --attention-backend HPC_ATTN”
“Enable the fused MoE backend with --moe-backend hpc”
agentic scaffolding 1
“--tool-call-parser hy_v3”
1 more mentioned or evaluated
inference quantization “AngelSlim covers quantization, low-bit methods, and speculative sampling.”
1 mentioned or evaluated
kernel & quantization library “The research team points to its AngelSlim toolkit. AngelSlim covers quantization, low-bit methods, and speculative sampling.”
human & real-world evaluation 1
“Hy3 has been refined through use by global developers and across Tencent's large-scale real-world business scenarios”
unfiled 6
“Hy3 is a hybrid fast-and-slow-thinking model built on a Mixture-of-Experts (MoE) architecture”
“through strengthened reinforcement learning and enhanced data quality and diversity”
“export VLLM_ROCM_USE_AITER_LINEAR=1”
“export VLLM_ROCM_USE_AITER_MHA=1”
“export VLLM_ROCM_USE_AITER_RMSNORM=1”
Not used 1
“set VLLM_ROCM_USE_AITER_MOE=0 to fall back to the Triton MoE kernel”
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| tencent.com/en-us/articles/2202386.html | official blog | Tencent | read |
| github.com/Tencent-Hunyuan/Hy3 | code repo | Tencent Hunyuan | read |
| huggingface.co/tencent/Hy3 | model card | Tencent Hunyuan | read |
| recipes.vllm.ai/tencent/Hy3 | vendor docs | vLLM | read |
| marktechpost.com/2026/07/06/tencent-releases-hy3-open-295b-moe-model | news | MarkTechPost | read |
| technode.com/2026/07/07/tencent-launches-hunyuan-hy3-integrates-model-a… | news | TechNode | read |
| gigazine.net/gsc_news/en/20260707-tencent-ai-hy3 | news | GIGAZINE | read |