Open Models Almanac
ModelsHy3

Tencent (Hunyuan) · usage rank #9

Hy3

What Hy3’s own documents say it is built from — every method with the sentence that describes it.

Parameters
295B / 21B active
Context
256K
Attention
full (GQA/MHA)GQA with QK-norm, every layer
Experts
192 routed + 1 shared, top-8
Input
text
Licence
Apache 2.0
Released
2026-07-06

Rank by open-weight tokens on OpenRouter, week of 2026-10-26.

Against the consensus recipe

Its documents state 6 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 6 of the 19 architecture features checked — 3 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_hy_v3.py, configuration_hy_v3.py) and config.json at revision a960ebc3da, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention in its codeadopted

64 query heads share 8 KV heads via repeat_kv in every attention layer.

show evidencenum_attention_heads = 64num_key_value_heads = 8
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_hy_v3.py · L207
Multi-head latent attention not in its codenot stated

Attention uses standard separate q/k/v projections sized by head_dim with no low-rank latent kv projection anywhere in the code.

Sliding-window attention not in its codenot stated

The model builder explicitly uses a full causal mask with no sliding window mechanism.

show evidence
# No sliding window opposed to mixtral
        causal_mask = create_causal_mask(
modeling_hy_v3.py · L489–490
Interleaved sliding-window and global attention not in its codenot stated

There is no layer_types or sliding_window_pattern config, and all attention layers use the same full causal mask.

Indexer-selected sparse attention (DeepSeek Sparse Attention) not in its codenot stated

No indexer module or top-k token selection mechanism exists in the attention code, and there are no index_topk-related config keys.

Linear-attention or state-space layers alongside full attention not in its codenot stated

All decoder layers use the same HYV3Attention softmax attention; there is no recurrent/linear mixer layer type.

show evidence
self.self_attn = HYV3Attention(config=config, layer_idx=layer_idx)
modeling_hy_v3.py · L372
Gated DeltaNet layers not in its codenot stated

No delta-rule recurrent state or decay gate implementation is present in the code.

Mamba-2 layers not in its codenot stated

No SSM/Mamba-2 selective state-space block or related config keys (mamba_num_heads, ssm_state_size) appear in this model.

Learnable attention sink not in its codenot stated

The attention forward computes a standard softmax over query-key scores with no extra learned sink logit column.

show evidence
attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
modeling_hy_v3.py · L190
Gated attention output not in its codenot stated

The attention output is passed directly to o_proj with no sigmoid gating multiplication before the output projection.

show evidence
attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)
modeling_hy_v3.py · L265–266
QK normalization code only in its codenot stated

Queries and keys are each passed through a per-head RMSNorm before RoPE and the attention dot product, and qk_norm is enabled in config.

show evidenceqk_norm = true
query_states = self.q_norm(query_states)
        key_states = self.k_norm(key_states)
modeling_hy_v3.py · L241–242
Partial RoPE not in its codenot stated

RoPE is computed over the full head_dim (dim = head_dim) and applied to the whole query/key vectors, with no partial_rotary_factor or…

show evidence

RoPE is computed over the full head_dim (dim = head_dim) and applied to the whole query/key vectors, with no partial_rotary_factor or rope_dim splitting.

dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads
modeling_hy_v3.py · L94
YaRN RoPE scaling not in its codenot stated

rope_parameters.rope_type is set to 'default', not yarn, so no YaRN scaling is applied.

show evidencerope_parameters = {"rope_theta": 11158840.0, "rope_type": "default"}
Layers without positional encoding (NoPE) not in its codenot stated

RoPE (via position_embeddings) is applied uniformly to every attention layer with no per-layer no-rope switch.

show evidence
query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
modeling_hy_v3.py · L245
Mixture of experts in its codeadopted

Feed-forward layers route tokens to 8 of 192 experts via a learned sigmoid router (HYV3TopKRouter) in all but the first dense layer.

show evidencenum_experts = 192num_experts_per_tok = 8
self.mlp = HYV3MoE(config) if config.mlp_layer_types[layer_idx] == "sparse" else HYV3MLP(config)
modeling_hy_v3.py · L373
Shared expert in its codeadopted

Each MoE layer includes a shared HYV3MLP expert sized by num_shared_experts=1 that every token passes through and adds to the routed output.

show evidencenum_shared_experts = 1
shared_intermediate = config.moe_intermediate_size * config.num_shared_experts
        self.shared_experts = HYV3MLP(config, intermediate_size=shared_intermediate)
modeling_hy_v3.py · L347–348
Auxiliary-loss-free load balancing (selection bias) code only in its codenot stated

The router adds e_score_correction_bias to the sigmoid routing scores only for top-k selection, not for weighting, matching…

show evidence

The router adds e_score_correction_bias to the sigmoid routing scores only for top-k selection, not for weighting, matching DeepSeek-V3-style bias balancing.

moe_router_enable_expert_bias = true
scores_for_choice = routing_weights + e_score_correction_bias
        _, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False)
        top_k_weights = routing_weights.gather(1, top_k_index)
modeling_hy_v3.py · L288–290
Multi-token prediction layers documents disagree not in its codeadopted

Config declares num_nextn_predict_layers=1, but the model explicitly ignores/skips those weights and builds no MTP module.

show evidencenum_nextn_predict_layers = 1
# Not supporting multi-token prediction (MTP) atm
    _keys_to_ignore_on_load_unexpected = [r"model\.layers\.80.*"]
modeling_hy_v3.py · L427–428
SwiGLU feed-forward code only in its codenot stated

Both the dense MLP and the MoE experts compute silu(gate(x)) * up(x) before the down projection.

show evidencehidden_act = "silu"
down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
modeling_hy_v3.py · L124

token mixer 1

softmax attention 1

grouped-query attention 1

Grouped-query attention core

“Attention Heads | 64 (GQA, 8 KV heads, head dim 128)”

channel mixer 4

mixture of experts 4

Mixture of experts core

“Hy3 is a hybrid fast-and-slow-thinking model built on a Mixture-of-Experts (MoE) architecture”

Sparse expert activation core

“192 routed experts (top-8) + 1 shared expert”

expert routing 1

Top-k expert routing core

“Number of Experts | 192 experts, top-8 activated”

shared experts 1

Shared expert core

“192 routed experts (top-8) + 1 shared expert”

prediction head 1

Multi-Token Prediction core

“Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model with 21B active parameters and 3.8B MTP layer parameters”

training stability 1

Anti-hallucination training used

“The anti-hallucination training reduces fabricated clauses and misquotes.”

data filtering 2

Data cleaning used

“we implemented fine-grained data cleaning and training constraints”

Evidence-based data cleaning and training constraints used

“Hy3 implements data cleaning and training constraints based on the principle of 'answering only when there is evidence, and clearly…”

filed at the root 1

Joint supervised fine-tuning and reinforcement learning used

“Joint SFT and RL improved coreference and constraint tracking.”

supervised fine-tuning 2

Supervised fine-tuning used

“Through joint optimization of SFT and RL”

Fine-tuning optional

“Hy3 provides a complete model finetuning pipeline”

reinforcement learning algorithm 2

Reinforcement learning used

“through strengthened reinforcement learning and enhanced data quality and diversity”

Group Relative Policy Optimization used

“Hy3 supports GRPO reinforcement learning training with verl”

decoding strategy 3

Speculative decoding core

“for speculative decoding”

EAGLE optional

“--speculative-algorithm EAGLE”

Speculative sampling optional

“AngelSlim supports a comprehensive suite of compression tools ... including ... speculative sampling”

reasoning control 3

Thinking mode default

“reasoning_effort: "no_think" (default, direct response), "low", "high" (deep chain-of-thought)”

Reasoning parser used

“--reasoning-parser hy_v3”

Chain-of-thought reasoning optional

“reasoning_effort: 'no_think' (default, direct response), 'low', 'high' (deep chain-of-thought)”

inference quantization 3

FP8 used

“Hy3-FP8 | FP8 quantized instruct model”

FP8 KV-cache quantization optional

“--kv-cache-dtype fp8_e4m3”

Quantization optional

“AngelSlim supports a comprehensive suite of compression tools ... including common quantization algorithms, low-bit quantization”

serving parallelism 1

TensorRT-LLM all-reduce backend used

“export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm”

inference kernel 3

Triton MoE kernel used

“fall back to the Triton MoE kernel”

HPC-Ops attention backend optional

“Enable the attention backend with --attention-backend HPC_ATTN”

HPC-Ops fused MoE backend optional

“Enable the fused MoE backend with --moe-backend hpc”

agentic scaffolding 1

Tool-call parser used

“--tool-call-parser hy_v3”

1 more mentioned or evaluated
Low-bit quantization mentioned

inference quantization “AngelSlim covers quantization, low-bit methods, and speculative sampling.”

1 mentioned or evaluated
AngelSlim mentioned

kernel & quantization library “The research team points to its AngelSlim toolkit. AngelSlim covers quantization, low-bit methods, and speculative sampling.”

human & real-world evaluation 1

Real-world feedback refinement used

“Hy3 has been refined through use by global developers and across Tencent's large-scale real-world business scenarios”

unfiled 6

Hybrid fast-and-slow-thinking model core

“Hy3 is a hybrid fast-and-slow-thinking model built on a Mixture-of-Experts (MoE) architecture”

Data quality and diversity enhancement used

“through strengthened reinforcement learning and enhanced data quality and diversity”

AITER Linear backend optional

“export VLLM_ROCM_USE_AITER_LINEAR=1”

AITER MHA backend optional

“export VLLM_ROCM_USE_AITER_MHA=1”

AITER RMSNorm backend optional

“export VLLM_ROCM_USE_AITER_RMSNORM=1”

Not used 1

AITER fused-MoE backend not used

“set VLLM_ROCM_USE_AITER_MOE=0 to fall back to the Triton MoE kernel”

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

↑