Tencent (Hunyuan) · usage rank #10 · released 2026-07-06

Hy3

What Hy3’s own documents say it is built from — every method with the sentence that describes it.

Curator’s note295B total / 21B active MoE (192 experts, top-8) + 3.8B MTP layer, 256K context, Apache 2.0. Weekly total merges the paid (3.33T) and :free (1.76T) variants. Distinct from the April 'hy3-preview'.

Rank by open-weight tokens on OpenRouter, week of 2026-09-21.

Against the consensus recipe

Its documents state 7 of the 15 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something the documents do not say, not something the model lacks.

Data curation
- Agentic data synthesis pipelinenot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- KL alignment lossnot in its documents
Model architecture
- Mixture of Expertscore
- Hybrid Attentionnot in its documents
- Multi-Token Predictioncore
Optimization
- Muonnot in its documents
- Expert Parallelismnot in its documents
- Cosine decaynot in its documents
Post-training
- Group Relative Policy Optimizationused
- Supervised fine-tuningused
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Configurable reasoning effortdefault
- Speculative decodingcore
- FP8 KV-cache quantizationoptional

Checked in its code

Its code shows 6 of the 19 architecture features checked — 3 of them are not stated in its documents.

Documents say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_hy_v3.py, configuration_hy_v3.py) and config.json at revision a960ebc3da, by anthropic/claude-sonnet-5 on 2026-09-25.

Feature	Code	Its documents	Evidence
Grouped-query attention	in its code	core	64 query heads share 8 key/value heads via repeat_kv in every attention layer num_attention_heads = 64 num_key_value_heads = 8 `self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads` modeling_hy_v3.py · L218
Multi-head latent attention	not in its code	not stated	Attention uses standard separate q/k/v projections sized by head counts, with no low-rank latent kv projection anywhere in the code
Sliding-window attention	not in its code	not stated	The model builds a plain causal mask with no sliding window mechanism, as noted explicitly in the code comment `# No sliding window opposed to mixtral causal_mask = create_causal_mask(` modeling_hy_v3.py · L500–501
Interleaved sliding-window and global attention	not in its code	not stated	There is only one attention layer type (HYV3Attention) applied uniformly with a full causal mask, no interleaving of local/global layers
Indexer-selected sparse attention (DeepSeek Sparse Attention)	not in its code	not stated	No indexer module or top-k token selection mechanism exists in the attention code; the config has no index_topk/index_n_heads keys
Linear-attention or state-space layers alongside full attention	not in its code	not stated	All decoder layers use the same HYV3Attention softmax attention module; there is no linear-attention or state-space mixer variant in the code `self.self_attn = HYV3Attention(config=config, layer_idx=layer_idx)` modeling_hy_v3.py · L383
Gated DeltaNet layers	not in its code	not stated	No gated delta-rule recurrent module is implemented anywhere in the file
Mamba-2 layers	not in its code	not stated	No Mamba-2/SSD selective state-space block is implemented; the config lacks mamba_num_heads or ssm_state_size
Learnable attention sink	not in its code	not stated	The softmax attention computation has no extra learned sink logit column added to the denominator `attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)` modeling_hy_v3.py · L201
Gated attention output	not in its code	not stated	The attention output is passed directly to o_proj with no sigmoid gating multiplication applied `attn_output = attn_output.reshape(*input_shape, -1).contiguous() attn_output = self.o_proj(attn_output)` modeling_hy_v3.py · L276–277
QK normalization code only	in its code	not stated	Per-head RMSNorm is applied to queries and keys before the rotary embedding and attention dot product, matching qk_norm: true in the config qk_norm = true `query_states = self.q_norm(query_states) key_states = self.k_norm(key_states)` modeling_hy_v3.py · L252–253
Partial RoPE	not in its code	not stated	RoPE is computed and applied over the full head dimension (dim = head_dim, cos/sin concatenated over full size) with no partial rotary factor or split `dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads` modeling_hy_v3.py · L97
YaRN RoPE scaling	not in its code	not stated	rope_parameters.rope_type is set to 'default', not YaRN, so no rope scaling is applied rope_parameters = {"rope_theta": 11158840.0, "rope_type": "default"}
Layers without positional encoding (NoPE)	not in its code	not stated	Rotary position embeddings are applied uniformly to every attention layer via the shared rotary_emb module with no per-layer skip logic `query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)` modeling_hy_v3.py · L256
Mixture of experts	in its code	core	Feed-forward layers (all but the first) route tokens to 8 of 192 experts via a learned sigmoid router num_experts = 192 num_experts_per_tok = 8 first_k_dense_replace = 1 `self.mlp = HYV3MoE(config) if config.mlp_layer_types[layer_idx] == "sparse" else HYV3MLP(config)` modeling_hy_v3.py · L384
Shared expert	in its code	core	Each MoE layer includes a shared expert MLP with nonzero intermediate size (num_shared_experts=1) added to every token's output num_shared_experts = 1 `shared_intermediate = config.moe_intermediate_size * config.num_shared_experts self.shared_experts = HYV3MLP(config, intermediate_size=shared_intermediate)` modeling_hy_v3.py · L358–359
Auxiliary-loss-free load balancing (selection bias) code only	in its code	not stated	The router adds an e_score_correction_bias to routing scores only for top-k selection, not for the output weighting, matching moe_router_enable_expert_bias: true moe_router_enable_expert_bias = true `scores_for_choice = routing_weights + e_score_correction_bias _, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False) top_k_weights = routing_weights.gather(1, top_k_index)` modeling_hy_v3.py · L299–301
Multi-token prediction layers documents disagree	not in its code	core	The config declares num_nextn_predict_layers: 1 but the code explicitly states MTP is not supported and skips loading those weights num_nextn_predict_layers = 1 `# Not supporting multi-token prediction (MTP) atm _keys_to_ignore_on_load_unexpected = [r"model\.layers\.80.*"]` modeling_hy_v3.py · L438–439
SwiGLU feed-forward code only	in its code	not stated	Both the dense MLP and expert MLPs compute silu(gate(x)) * up(x), i.e. SwiGLU, with hidden_act set to silu hidden_act = "silu" `down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))` modeling_hy_v3.py · L135 `current_hidden_states = self.act_fn(gate) * up` modeling_hy_v3.py · L342

model architecture 6

token mixer 1

softmax attention 1

grouped-query attention 1

Grouped-query attention core

channel mixer 4

inference & serving 15

decoding strategy 3

Speculative decoding core

EAGLE optional

Speculative sampling optional

reasoning control 3

Configurable reasoning effort default

Reasoning parser used

Chain-of-thought reasoning optional

inference quantization 4

FP8 used

FP8 KV-cache quantization optional

Quantization optional

Low-bit quantization mentioned

serving parallelism 1

TensorRT-LLM all-reduce backend used

inference kernel 3

Triton MoE kernel used

HPC-Ops attention backend optional

HPC-Ops fused MoE backend optional

evaluation 1

human & real-world evaluation 1

Real-world feedback refinement used

unfiled 6

Hybrid fast-and-slow-thinking model core

Data quality and diversity enhancement used

AITER Linear backend optional

AITER MHA backend optional

AITER RMSNorm backend optional

AITER fused-MoE backend not used

Sources

The curated document list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

Document	Kind	Publisher	Status
tencent.com/en-us/articles/2202386.html	official blog	Tencent	read
github.com/Tencent-Hunyuan/Hy3	code repo	Tencent Hunyuan	read
huggingface.co/tencent/Hy3	model card	Tencent Hunyuan	read
recipes.vllm.ai/tencent/Hy3	vendor docs	vLLM	read
marktechpost.com/2026/07/06/tencent-releases-hy3-open-295b-moe-model	news	MarkTechPost	read
technode.com/2026/07/07/tencent-launches-hunyuan-hy3-integrates-model-a…	news	TechNode	read
gigazine.net/gsc_news/en/20260707-tencent-ai-hy3	news	GIGAZINE	read

Hy3

Its documents state 7 of the 15 methods the field agrees on.

Data curation

Training objective

Model architecture

Optimization

Post-training

Inference & serving

Its code shows 6 of the 19 architecture features checked — 3 of them are not stated in its documents.

model architecture 6

token mixer 1

softmax attention 1

grouped-query attention 1

channel mixer 4

mixture of experts 4

expert routing 2

shared experts 1

prediction head 1

optimization 1

training stability 1

data curation 2

data filtering 2

post-training 6

filed at the root 1

supervised fine-tuning 2

reinforcement learning algorithm 3