Model techniques map
ModelsKimi K3

Moonshot AI · released 2026-07-16

Kimi K3

What Kimi K3’s own documents say it is built from — every method with the sentence that describes it.

OpenRouterHugging Face

Curator’s noteRe-crawled 2026-07-27: the promised open-weights drop landed, so the first-party set below replaces what was announcement coverage only. 2.8T total / 104B activated (16 of 896 routed experts via Stable LatentMoE), 1M context, native vision, built on Kimi Delta Attention + Attention Residuals; weights under the Modified-MIT 'Kimi K3 License' (MaaS attribution clause). The pre-release note here estimated ~50B active — the technical report says 104B, so prefer the report. Both watch URLs resolved: github.com/MoonshotAI/Kimi-K3 and huggingface.co/moonshotai/Kimi-K3 now exist.

Against the consensus recipe

Its documents state 8 of the 15 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something the documents do not say, not something the model lacks.

  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 8 of the 19 architecture features checked — 4 of them are not stated in its documents.

Documents say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from the model repository’s own code (modeling_kimi_k3.py, configuration_kimi_k3.py, modeling_kimi_linear.py) and config.json at revision f831ab6681, by anthropic/claude-sonnet-5 on 2026-09-25.

FeatureCodeIts documentsEvidence
Grouped-query attention not in its codenot stated

The MLA attention module sets num_key_value_heads equal to num_attention_heads (both 96), so there is no query-group sharing of fewer KV heads, and attention is MLA rather than GQA anyway.

text_config.num_attention_heads = 96text_config.num_key_value_heads = 96
self.num_key_value_heads = config.num_key_value_heads
        self.num_key_value_groups = self.num_heads // self.num_key_value_heads
modeling_kimi_linear.py · L346–347
Multi-head latent attention code only in its codenot stated

KimiMLAAttention down-projects hidden states into a shared kv_lora_rank latent (with a decoupled RoPE key dimension qk_rope_head_dim) and up-projects per head via kv_b_proj, DeepSeek-V2/V3 style.

text_config.kv_lora_rank = 512text_config.qk_rope_head_dim = 64text_config.q_lora_rank = 1536
self.kv_a_proj_with_mqa = nn.Linear(
            self.hidden_size,
            self.kv_lora_rank + self.qk_rope_head_dim,
            bias=False,
        )
        self.kv_a_layernorm = KimiRMSNorm(self.kv_lora_rank)
modeling_kimi_linear.py · L378–383
compressed_kv = self.kv_a_proj_with_mqa(hidden_states)
        k_pass, k_rot = torch.split(
            compressed_kv, [self.kv_lora_rank, self.qk_rope_head_dim], dim=-1)
modeling_kimi_linear.py · L426–428
Sliding-window attention not in its codenot stated

No sliding_window config key or windowed masking code exists; the only attention pattern switch is between full (MLA) attention layers and linear KDA layers.

Interleaved sliding-window and global attention not in its codenot stated

There is no interleaving of sliding-window and global attention layers; the fixed interleaving present is between full-attention (MLA) and linear-attention (KDA) layers, not local-window vs global attention.

Indexer-selected sparse attention (DeepSeek Sparse Attention) not in its codenot stated

No indexer-based top-k token selection mechanism (index_topk, index_n_heads, etc.) appears anywhere in the config or code.

Linear-attention or state-space layers alongside full attention in its code core

The text_config's linear_attn_config defines kda_layers (linear-attention KDA layers) interleaved with full_attn_layers (full MLA softmax attention layers) across the 93 hidden layers.

text_config.linear_attn_config.kda_layers = [1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 14, 15, 17, 18, 19, 21, 22, 23, 25, 26, 27,…text_config.linear_attn_config.full_attn_layers = [4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80…
if config.is_kda_layer(layer_idx):
            self.is_linear_attn = True
            self.self_attn = KimiDeltaAttention(
                config=config, layer_idx=layer_idx)
        elif config.is_mla:
            self.is_linear_attn = False
            self.self_attn = KimiMLAAttention(
                config=config, layer_idx=layer_idx)
modeling_kimi_linear.py · L883–890
Gated DeltaNet layers in its code used

KimiDeltaAttention implements Kimi Delta Attention (a Gated DeltaNet variant) with a delta-rule recurrent update, decay gate (A_log/dt_bias), and short convolutions on q/k/v, applied to the kda_layers.

text_config.linear_attn_config.num_heads = 96text_config.linear_attn_config.short_conv_kernel_size = 4
self.A_log = torch.nn.Parameter(torch.log(torch.empty(
            self.num_heads, dtype=torch.float32).uniform_(1, 16)))

        self.f_a_proj = nn.Linear(self.hidden_size, self.head_dim, bias=False)
        self.f_b_proj = nn.Linear(self.head_dim, projection_size, bias=False)
modeling_kimi_linear.py · L520–524
o, recurrent_state = chunk_kda(
                q=q,
                k=k,
                v=v,
                g=g,
                beta=beta,
                A_log=self.A_log,
                dt_bias=self.dt_bias,
modeling_kimi_linear.py · L610–617
Mamba-2 layers not in its codenot stated

No Mamba-2/SSD selective state-space block (mamba_num_heads, ssm_state_size) exists in the code or config; the linear-attention layers use KDA (delta-rule), not Mamba-2.

Learnable attention sink not in its codenot stated

No per-head learned sink logit is added to the attention softmax denominator anywhere in the eager or MLA attention implementations.

Gated attention output code only in its codenot stated

In KimiMLAAttention, when mla_use_output_gate is true, a sigmoid gate computed from hidden_states (g_proj) multiplies the attention output before the final projection.

text_config.mla_use_output_gate = true
if self.use_output_gate:
            g = self.g_proj(hidden_states).sigmoid()
            attn_output = attn_output * g
        attn_output = self.o_proj(attn_output)
modeling_kimi_linear.py · L470–473
QK normalization not in its codenot stated

KimiMLAAttention has no query/key RMSNorm or LayerNorm applied before the attention dot product (only kv_a_layernorm and q_a_layernorm normalize the compressed latents, not per-head q/k before attention).

Partial RoPE code only in its codenot stated

MLA queries/keys are split into a nope portion (qk_nope_head_dim=128) and a rope portion (qk_rope_head_dim=64), with only the rope portion carrying positional info via the decoupled key, and mla_use_nope confirms this partial scheme is active.

text_config.qk_nope_head_dim = 128text_config.qk_rope_head_dim = 64text_config.mla_use_nope = true
q_pass, q_rot = torch.split(
            q_states, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
modeling_kimi_linear.py · L423–424
k_pass = self.kv_b_proj(self.kv_a_layernorm(
            k_pass)).view(key_shape).transpose(1, 2)
        k_pass, value_states = torch.split(
            k_pass, [self.qk_nope_head_dim, self.v_head_dim], dim=-1)
modeling_kimi_linear.py · L430–433
YaRN RoPE scaling code unclearnot stated

The config exposes a rope_scaling attribute in KimiLinearConfig defaulting to None and it is not present as a key in the given text_config, and no rotary_emb / RoPE application code is shown (self.rotary_emb is set to None and never used), so YaRN scaling cannot be confirmed as active.

self.rotary_emb = None
modeling_kimi_linear.py · L403
Layers without positional encoding (NoPE) documents disagree not in its code core

All full-attention (MLA) layers use mla_use_nope=True uniformly, meaning nope is applied to every softmax-attention layer's designated portion rather than some layers having positional encoding and others not; there is no layer-varying NoPE pattern.

text_config.mla_use_nope = true
self.use_nope = config.mla_use_nope
            self.scaling = self.q_head_dim ** (-0.5)
modeling_kimi_linear.py · L358–359
Mixture of experts in its code core

KimiSparseMoeBlock routes tokens among num_experts=896 expert MLPs choosing num_experts_per_token=16 via a learned KimiMoEGate router, applied to layers past first_k_dense_replace.

text_config.num_experts = 896text_config.num_experts_per_token = 16text_config.first_k_dense_replace = 1
self.experts = nn.ModuleList(
            [
                KimiBlockSparseMLP(
                    config,
                    hidden_size=self.moe_hidden_size,
                    intermediate_size=config.moe_intermediate_size,
                )
                for _ in range(config.num_experts)
            ],
        )
        self.gate = KimiMoEGate(config)
modeling_kimi_linear.py · L786–796
Shared expert code only in its codenot stated

KimiSparseMoeBlock builds a shared_experts KimiMLP sized by num_shared_experts=2 that is added to every token's output alongside the routed experts.

text_config.num_shared_experts = 2
if config.num_shared_experts is not None:
            intermediate_size = config.moe_intermediate_size * config.num_shared_experts
            self.shared_experts = KimiMLP(
                config=config, intermediate_size=intermediate_size,
            )
modeling_kimi_linear.py · L797–801
if self.config.num_shared_experts is not None:
            y = y + self.shared_experts(identity)
modeling_kimi_linear.py · L836–837
Auxiliary-loss-free load balancing (selection bias) in its code core

KimiMoEGate adds a learned e_score_correction_bias to the routing scores solely for top-k expert selection (topk_method=noaux_tc), matching DeepSeek-V3's aux-loss-free bias mechanism.

text_config.topk_method = "noaux_tc"
self.e_score_correction_bias = nn.Parameter(
            torch.empty(self.num_experts),
        )
modeling_kimi_linear.py · L693–695
scores_for_choice = scores + self.e_score_correction_bias.unsqueeze(0)
modeling_kimi_linear.py · L723
Multi-token prediction layers not in its codenot stated

num_nextn_predict_layers is set to 0 in this config, so no multi-token-prediction modules are built even though the config field exists.

text_config.num_nextn_predict_layers = 0
SwiGLU feed-forward not in its codenot stated

hidden_act is set to "situ" and the MLP/expert modules use the custom SituAndMul activation (beta*tanh(gate/beta)*sigmoid(gate)*up), not a SiLU-gated SwiGLU.

text_config.hidden_act = "situ"
situ_a = self.beta * torch.tanh(gate / self.beta) * torch.sigmoid(gate)
        if self.linear_beta is not None:
            up = self.linear_beta * torch.tanh(up / self.linear_beta)
        return (situ_a * up).to(x.dtype)
modeling_kimi_linear.py · L79–82

model architecture 40

token mixer 10

Factorized spatial-temporal video attention with temporal pooling used

softmax attention 1

multi-head latent attention 1

Gated Multi-Head Latent Attention core

hybrid layer stacking 1

Hybrid Attention core

channel mixer 17

dense feed-forward network 2

SiTU-GLU core
SiTU used

positional encoding 1

No Position Encoding core

normalization & residual 4

Attention Residuals (AttnRes) core
Block AttnRes core
Two-Phase Block AttnRes Schedule core
Full Attention Residuals mentioned

multimodal architecture 7

Single shared multimodal backbone core
2×2 pixel-shuffle downsampling used
Decoupled Encoder Process (DEP) used
Dual-format coordinate supervision used
From-scratch vision-encoder training with next-token prediction used
MoonViT-V2 used
Multimodal input used

context capacity 1

Progressive context extension used

optimization 33

filed at the root 1

Updated hyperparameter scaling law used

optimizer 3

Muon used
Per-Head Muon used
Peer-to-peer shard retrieval for Muon orthogonalization used

learning-rate schedule 2

Cosine decay default
Warmup-Stable-Decay not used

training precision 2

Block-wise FP8 activation quantization with offload used
FP32 attention-output retention used

quantization-aware training 3

Quantization-Aware Training used
MXFP4 Weights with MXFP8 Activations used
MXFP4 Quantization-Aware Post-Training used

training stability 3

Per-token regularization for off-policy RL used
Soft dropping used
Weight clipping used

training parallelism 12

MoonEP core
Redundant-Expert Capacity Reservation core
Sequence Parallelism for Activations core
Expert Parallelism used
Cache-Based Pipeline Communication used
Dynamic Context Parallelism for Large Multimodal Samples used
Fully Balanced Expert-Parallel Training used
GPU Planning Kernel for Redundant Expert Migration used
KDA Context Parallelism used
Pipeline ZeRO-2 Gradient Sharding with CPU Offloading used
pipeline-bubble scheduling of ViT computation used
SM-Level Context Parallelism used

training runtime 7

Composable activation storage policies core
Unified pluggable activation storage manager core
Cross-rank remote activation offloading used
Double-buffered chunked streaming of reference-model weights used
Element-wise activation recomputation used
Policy-model gradient-buffer reuse for reference-model weights used
Post-iteration NVMe offloading of training states used

data curation 12

data sourcing 1

Agentic knowledge graph construction used

data filtering 2

Domain filtering with heuristics, classifier scoring, and deduplication used
Long-context data cleaning pipeline used

synthetic data 7

Corpus rephrasing with fidelity verification used
Knowledge-graph-guided task synthesis used
Long-context data synthesis by permutation and concatenation used
Multi-stage verification with human-in-the-loop annotation used
Programmatic multimodal data generation used
Sandboxed visual reasoning with a Python interpreter used
Synthetic trajectory generation with domain-specialized models used

data mixture & curriculum 1

Long-context data upsampling used

tokenization 1

XTML (eXtensible Token Markup Language) core

post-training 17

filed at the root 1

Three-stage post-training with multi-teacher on-policy distillation core

supervised fine-tuning 1

Supervised fine-tuning used

reinforcement learning algorithm 2

Multi-Domain, Multi-Effort Reinforcement Learning core
Domain-Specialized RL Experts used

reward modelling 3

Agentic Generative Reward Model used
Public and hidden verifier pairing used
Web-development reward with deterministic checks and model judging used

preference optimization 1

Budget-based verbosity control used

policy distillation 2

Multi-Teacher On-Policy Distillation core
Per-Token On-Policy Distillation Reward used

rollout & RL infrastructure 3

Partial rollout used
Co-located RL training used
Co-located long-context agentic RL system used

agentic post-training 4

Training on verifiable problems in agentic environments core
Autonomous Execution Tasks (AET) used
Mock applications for personal-assistant reinforcement learning used
Unified white-box reinforcement-learning environment used

inference & serving 54

decoding strategy 2

Fused recurrent replay kernel core
EAGLE-3-style draft-model fine-tuning used

reasoning control 8

Always-on thinking mode core
Configurable reasoning effort default
Chain-of-thought reasoning default
Maximum thinking effort default
Natural-language reasoning-effort option message used
Per-problem reasoning-budget control used
stage-wise curriculum over reasoning-effort budget multiplier used
Thinking mode selection via generation prefix used

KV cache management 9

Fine-grained prefix hashing core
KDA-aware prefix-cache management core
Projected-input caching for speculative KDA rollback core
Shared-free-list cache allocation core
Sparse hash-aligned KDA recurrent-state checkpoints core
Unified paged cache layout for KDA states and MLA KV core
Write-back external KV-cache policy core
KDA with prefill cache used
KV-cache-aware placement of one-shot option messages used

inference quantization 3

Offline weight-layout permutation core
MXFP4 weight quantization used
MXFP8 activation quantization used

serving parallelism 2

Zero-copy fused token permutation and unpermutation core
Token migration for balanced expert placement unclear

inference scheduling 6

Cross-group pinning of cache-hit blocks core
Dual-cluster consistent-hash failover for cache affinity core
Prefix-cache-aware session affinity scheduling core
Request-class resource-budget admission control core
Runtime-signal-based rollout concurrency auto-throttling core
Workload-aware routed-expert GEMM scheduling core

inference kernel 9

Fused AttnRes merge and RMSNorm kernel core
Fused latent down-projection and MoE-router GEMM core
GPU planning kernel for online expert placement core
Separate-stream shared-expert GEMM overlap core
Side-stream overlap for inter-block attention-residual computation core
Synchronization-free static-shape MoE execution core
WarpDecode token-centric routed-expert decoding kernel core
FlashKDA used
KDA algorithm–system co-design used

context management 2

Preserved thinking history mode core
Context compaction used

agentic scaffolding 13

Natural-language in-context option instructions core
Sandbox snapshots core
Sandbox state forking core
AgentENV used
Concurrent subagent orchestration used
Copy-on-write memory and page-cache optimization used
Dynamic tool loading used
Indexed parallel tool calls used
OverlayBD shared-image sandbox launch stack used
Typed tool arguments used
Vision in the loop used
XTML chat template used
Agentic tool calling mentioned

software implementation 7

kernel & quantization library 3

MiniTriton used
MiniTriton dual-mode tensor library used
Triton mentioned

agent product 1

OpenClaw used

infrastructure service 3

Incremental sandbox checkpointing and resumption core
Sandbox pause and resume core
Mooncake used

evaluation 10

filed at the root 6

End-to-end exploit development evaluation used
Prompt engineering to elicit answers used
Sandboxed GPU kernel optimization evaluation used
Trajectory analysis used
Two-tier cybersecurity capability evaluation used
Vulnerability discovery with proof-of-concept evaluation used

benchmark 2

Living in-house benchmark suite core
GPU kernel optimization task suite used

judge 1

Mandatory agentic judge protocol used

human & real-world evaluation 1

Blind side-by-side expert evaluation used

other 2

filed at the root 2

Reward-Hacking Detection System used
Score-versus-Cost Efficiency Comparison used

unfiled 8

native multimodal training from the start of training core
pre-training scaling core
architectural innovation used
autonomous chip design and verification with open-source EDA tools used
iterative refinement of a deliverable via web search and terminal queries used
likelihood-based acceptance-rate loss (LK loss) used
tool augmentation during evaluation used
tool-augmented benchmark evaluation (with and without tools) used

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated document list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

DocumentKindPublisherStatus
kimi.com/blog/kimi-k3official blogMoonshot AIread
github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdftechnical reportKimi Teamread
huggingface.co/moonshotai/Kimi-K3model cardMoonshot AIread
github.com/MoonshotAI/Kimi-K3code repoMoonshot AIread
interconnects.ai/p/kimi-k3-the-open-weights-escalationthird party analysisInterconnects (Nathan Lambert)read
simonwillison.net/2026/Jul/16/kimi-k3third party analysisSimon Willisonread
venturebeat.com/technology/chinas-moonshot-ai-releases-kimi-k3-the-larg…newsVentureBeatnot fetched
tomshardware.com/tech-industry/artificial-intelligence/moonshot-release…newsTom's Hardwareread
cnbc.com/2026/07/17/moonshot-ai-kimi-k3-model-openai-anthropic-china.ht…newsCNBCread
huggingface.co/blog/ResterChed/kimi-k3-model-overview-mxfp4-quantizatio…third party analysisHugging Face community blogread
openrouter.ai/moonshotai/kimi-k3vendor docsOpenRouterread