Open Models Almanac
ModelsKimi K3

Moonshot AI

Kimi K3

What Kimi K3’s own documents say it is built from — every method with the sentence that describes it.

Parameters
2.8T / 104B active
Context
1M
Attention
linear hybridKimi Delta Attention (KDA) linear layers + MLA layers, Attention Residuals
Experts
896 routed + 2 shared, top-16 (Stable LatentMoE)
Input
text, image
Licence
Modified MIT (Kimi K3 License)
Released
2026-07-16

Against the consensus recipe

Its documents state 7 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 8 of the 19 architecture features checked — 4 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from the model repository’s own code (modeling_kimi_k3.py, configuration_kimi_k3.py, modeling_kimi_linear.py) and config.json at revision f831ab6681, by anthropic/claude-sonnet-5 on 2026-09-25.

FeatureCodeIts documentsVerdict
Grouped-query attention not in its codenot stated

The MLA attention module sets num_key_value_heads equal to num_attention_heads (both 96), so there is no query-group sharing of fewer KV…

show evidence

The MLA attention module sets num_key_value_heads equal to num_attention_heads (both 96), so there is no query-group sharing of fewer KV heads, and attention is MLA rather than GQA anyway.

text_config.num_attention_heads = 96text_config.num_key_value_heads = 96
self.num_key_value_heads = config.num_key_value_heads
        self.num_key_value_groups = self.num_heads // self.num_key_value_heads
modeling_kimi_linear.py · L346–347
Multi-head latent attention code only in its codenot stated

KimiMLAAttention down-projects hidden states into a shared kv_lora_rank latent (with a decoupled RoPE key dimension qk_rope_head_dim) and…

show evidence

KimiMLAAttention down-projects hidden states into a shared kv_lora_rank latent (with a decoupled RoPE key dimension qk_rope_head_dim) and up-projects per head via kv_b_proj, DeepSeek-V2/V3 style.

text_config.kv_lora_rank = 512text_config.qk_rope_head_dim = 64text_config.q_lora_rank = 1536
self.kv_a_proj_with_mqa = nn.Linear(
            self.hidden_size,
            self.kv_lora_rank + self.qk_rope_head_dim,
            bias=False,
        )
        self.kv_a_layernorm = KimiRMSNorm(self.kv_lora_rank)
modeling_kimi_linear.py · L378–383
compressed_kv = self.kv_a_proj_with_mqa(hidden_states)
        k_pass, k_rot = torch.split(
            compressed_kv, [self.kv_lora_rank, self.qk_rope_head_dim], dim=-1)
modeling_kimi_linear.py · L426–428
Sliding-window attention not in its codenot stated

No sliding_window config key or windowed masking code exists; the only attention pattern switch is between full (MLA) attention layers and…

show the whole verdict

No sliding_window config key or windowed masking code exists; the only attention pattern switch is between full (MLA) attention layers and linear KDA layers.

Interleaved sliding-window and global attention not in its codenot stated

There is no interleaving of sliding-window and global attention layers; the fixed interleaving present is between full-attention (MLA) and…

show the whole verdict

There is no interleaving of sliding-window and global attention layers; the fixed interleaving present is between full-attention (MLA) and linear-attention (KDA) layers, not local-window vs global attention.

Indexer-selected sparse attention (DeepSeek Sparse Attention) not in its codenot stated

No indexer-based top-k token selection mechanism (index_topk, index_n_heads, etc.) appears anywhere in the config or code.

Linear-attention or state-space layers alongside full attention in its codeadopted

The text_config's linear_attn_config defines kda_layers (linear-attention KDA layers) interleaved with full_attn_layers (full MLA softmax…

show evidence

The text_config's linear_attn_config defines kda_layers (linear-attention KDA layers) interleaved with full_attn_layers (full MLA softmax attention layers) across the 93 hidden layers.

text_config.linear_attn_config.kda_layers = [1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 14, 15, 17, 18, 19, 21, 22, 23, 25, 26, 27,…text_config.linear_attn_config.full_attn_layers = [4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80…
if config.is_kda_layer(layer_idx):
            self.is_linear_attn = True
            self.self_attn = KimiDeltaAttention(
                config=config, layer_idx=layer_idx)
        elif config.is_mla:
            self.is_linear_attn = False
            self.self_attn = KimiMLAAttention(
                config=config, layer_idx=layer_idx)
modeling_kimi_linear.py · L883–890
Gated DeltaNet layers in its codeadopted

KimiDeltaAttention implements Kimi Delta Attention (a Gated DeltaNet variant) with a delta-rule recurrent update, decay gate…

show evidence

KimiDeltaAttention implements Kimi Delta Attention (a Gated DeltaNet variant) with a delta-rule recurrent update, decay gate (A_log/dt_bias), and short convolutions on q/k/v, applied to the kda_layers.

text_config.linear_attn_config.num_heads = 96text_config.linear_attn_config.short_conv_kernel_size = 4
self.A_log = torch.nn.Parameter(torch.log(torch.empty(
            self.num_heads, dtype=torch.float32).uniform_(1, 16)))

        self.f_a_proj = nn.Linear(self.hidden_size, self.head_dim, bias=False)
        self.f_b_proj = nn.Linear(self.head_dim, projection_size, bias=False)
modeling_kimi_linear.py · L520–524
o, recurrent_state = chunk_kda(
                q=q,
                k=k,
                v=v,
                g=g,
                beta=beta,
                A_log=self.A_log,
                dt_bias=self.dt_bias,
modeling_kimi_linear.py · L610–617
Mamba-2 layers not in its codenot stated

No Mamba-2/SSD selective state-space block (mamba_num_heads, ssm_state_size) exists in the code or config; the linear-attention layers use…

show the whole verdict

No Mamba-2/SSD selective state-space block (mamba_num_heads, ssm_state_size) exists in the code or config; the linear-attention layers use KDA (delta-rule), not Mamba-2.

Learnable attention sink not in its codenot stated

No per-head learned sink logit is added to the attention softmax denominator anywhere in the eager or MLA attention implementations.

Gated attention output code only in its codenot stated

In KimiMLAAttention, when mla_use_output_gate is true, a sigmoid gate computed from hidden_states (g_proj) multiplies the attention output…

show evidence

In KimiMLAAttention, when mla_use_output_gate is true, a sigmoid gate computed from hidden_states (g_proj) multiplies the attention output before the final projection.

text_config.mla_use_output_gate = true
if self.use_output_gate:
            g = self.g_proj(hidden_states).sigmoid()
            attn_output = attn_output * g
        attn_output = self.o_proj(attn_output)
modeling_kimi_linear.py · L470–473
QK normalization not in its codenot stated

KimiMLAAttention has no query/key RMSNorm or LayerNorm applied before the attention dot product (only kv_a_layernorm and q_a_layernorm…

show the whole verdict

KimiMLAAttention has no query/key RMSNorm or LayerNorm applied before the attention dot product (only kv_a_layernorm and q_a_layernorm normalize the compressed latents, not per-head q/k before attention).

Partial RoPE code only in its codenot stated

MLA queries/keys are split into a nope portion (qk_nope_head_dim=128) and a rope portion (qk_rope_head_dim=64), with only the rope portion…

show evidence

MLA queries/keys are split into a nope portion (qk_nope_head_dim=128) and a rope portion (qk_rope_head_dim=64), with only the rope portion carrying positional info via the decoupled key, and mla_use_nope confirms this partial scheme is active.

text_config.qk_nope_head_dim = 128text_config.qk_rope_head_dim = 64text_config.mla_use_nope = true
q_pass, q_rot = torch.split(
            q_states, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
modeling_kimi_linear.py · L423–424
k_pass = self.kv_b_proj(self.kv_a_layernorm(
            k_pass)).view(key_shape).transpose(1, 2)
        k_pass, value_states = torch.split(
            k_pass, [self.qk_nope_head_dim, self.v_head_dim], dim=-1)
modeling_kimi_linear.py · L430–433
YaRN RoPE scaling code unclearnot stated

The config exposes a rope_scaling attribute in KimiLinearConfig defaulting to None and it is not present as a key in the given…

show evidence

The config exposes a rope_scaling attribute in KimiLinearConfig defaulting to None and it is not present as a key in the given text_config, and no rotary_emb / RoPE application code is shown (self.rotary_emb is set to None and never used), so YaRN scaling cannot be confirmed as active.

self.rotary_emb = None
modeling_kimi_linear.py · L403
Layers without positional encoding (NoPE) documents disagree not in its codeadopted

All full-attention (MLA) layers use mla_use_nope=True uniformly, meaning nope is applied to every softmax-attention layer's designated…

show evidence

All full-attention (MLA) layers use mla_use_nope=True uniformly, meaning nope is applied to every softmax-attention layer's designated portion rather than some layers having positional encoding and others not; there is no layer-varying NoPE pattern.

text_config.mla_use_nope = true
self.use_nope = config.mla_use_nope
            self.scaling = self.q_head_dim ** (-0.5)
modeling_kimi_linear.py · L358–359
Mixture of experts in its codeadopted

KimiSparseMoeBlock routes tokens among num_experts=896 expert MLPs choosing num_experts_per_token=16 via a learned KimiMoEGate router…

show evidence

KimiSparseMoeBlock routes tokens among num_experts=896 expert MLPs choosing num_experts_per_token=16 via a learned KimiMoEGate router, applied to layers past first_k_dense_replace.

text_config.num_experts = 896text_config.num_experts_per_token = 16text_config.first_k_dense_replace = 1
self.experts = nn.ModuleList(
            [
                KimiBlockSparseMLP(
                    config,
                    hidden_size=self.moe_hidden_size,
                    intermediate_size=config.moe_intermediate_size,
                )
                for _ in range(config.num_experts)
            ],
        )
        self.gate = KimiMoEGate(config)
modeling_kimi_linear.py · L786–796
Shared expert code only in its codenot stated

KimiSparseMoeBlock builds a shared_experts KimiMLP sized by num_shared_experts=2 that is added to every token's output alongside the…

show evidence

KimiSparseMoeBlock builds a shared_experts KimiMLP sized by num_shared_experts=2 that is added to every token's output alongside the routed experts.

text_config.num_shared_experts = 2
if config.num_shared_experts is not None:
            intermediate_size = config.moe_intermediate_size * config.num_shared_experts
            self.shared_experts = KimiMLP(
                config=config, intermediate_size=intermediate_size,
            )
modeling_kimi_linear.py · L797–801
if self.config.num_shared_experts is not None:
            y = y + self.shared_experts(identity)
modeling_kimi_linear.py · L836–837
Auxiliary-loss-free load balancing (selection bias) in its codeadopted

KimiMoEGate adds a learned e_score_correction_bias to the routing scores solely for top-k expert selection (topk_method=noaux_tc)…

show evidence

KimiMoEGate adds a learned e_score_correction_bias to the routing scores solely for top-k expert selection (topk_method=noaux_tc), matching DeepSeek-V3's aux-loss-free bias mechanism.

text_config.topk_method = "noaux_tc"
self.e_score_correction_bias = nn.Parameter(
            torch.empty(self.num_experts),
        )
modeling_kimi_linear.py · L693–695
scores_for_choice = scores + self.e_score_correction_bias.unsqueeze(0)
modeling_kimi_linear.py · L723
Multi-token prediction layers not in its codenot stated

num_nextn_predict_layers is set to 0 in this config, so no multi-token-prediction modules are built even though the config field exists.

show evidencetext_config.num_nextn_predict_layers = 0
SwiGLU feed-forward not in its codenot stated

hidden_act is set to "situ" and the MLP/expert modules use the custom SituAndMul activation (beta*tanh(gate/beta)*sigmoid(gate)*up), not a…

show evidence

hidden_act is set to "situ" and the MLP/expert modules use the custom SituAndMul activation (beta*tanh(gate/beta)*sigmoid(gate)*up), not a SiLU-gated SwiGLU.

text_config.hidden_act = "situ"
situ_a = self.beta * torch.tanh(gate / self.beta) * torch.sigmoid(gate)
        if self.linear_beta is not None:
            up = self.linear_beta * torch.tanh(up / self.linear_beta)
        return (situ_a * up).to(x.dtype)
modeling_kimi_linear.py · L79–82

token mixer 10

Factorized spatial-temporal attention used

“attention is factorized into intra-frame spatial and inter-frame temporal passes, and temporal pooling further compresses tokens along the…”

softmax attention 1

multi-head latent attention 1

Gated Multi-head Latent Attention core

“MLA caches ct and reconstructs the content keys and values through learned up-projections during attention computation.”

linear attention & state space 7

gated delta network 7

Kimi Delta Attention core

“KDA extends the delta-rule recurrence [105, 138] with a channel-wise forget gate [63].”

KDA core

“Its architecture uses KDA and Attention Residuals for computational efficiency.”

Hybrid linear attention mechanism core

“A hybrid linear attention mechanism that replaces standard quadratic attention in a subset of layers.”

Kimi Delta Attention–Attention Residuals architecture core

“Kimi K3 is built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes)”

Gated DeltaNet used

“similar to the Gated DeltaNet used for Olmo Hybrid”

Input-dependent full-rank output gate used

“Kimi K3 changes KDA’s output gate from the low-rank parameterization used by Kimi Linear [63] to an input-dependent full-rank projection.”

Lower-bounded scaled-sigmoid log-decay used

“Kimi K3 instead uses a scaled sigmoid to bound the log-decay from below”

hybrid layer stacking 1

Layerwise Hybrid Linear and Global Attention core

“Kimi K3 uses a layerwise hybrid of linear and global attention, combining KDA [63] with Gated MLA.”

channel mixer 18

dense feed-forward network 2

SiTU-GLU core

“we propose Sigmoid Tanh Unit GLU (SiTU-GLU)”

Sigmoid Tanh Unit (SiTU) used

“Sigmoid Tanh Unit (SiTU): Custom activation function replacing standard GeLU/SwiGLU”

mixture of experts 16

Mixture of experts core

“We have also scaled up Mixture of Experts (MoE) sparsity, effectively activating 16 out of 896 experts when paired with a Stable LatentMoE…”

Sparse expert activation core

“activates just 16 of its 896 experts per token”

Dispatch recomputation core

“during the backward pass, the input of the group GEMM is recovered by recomputing dispatch.”

Output-independent MoE gradient reformulation core

“we rewrite this gradient through a mathematical transformation into a form that depends only on the intermediate activation act_output and…”

expert routing 2

Top-k routing with frozen bias default

“at deployment, routing is a fixed Top- k selection with a frozen bias, and no quantile”

Latent-space routing used

“Key components include latent-space routing, Quantile Balancing for load management, and soft dropping for overflow tokens.”

expert load balancing 7

Auxiliary-loss-free load balancing core

“Unlike auxiliary-loss-based routing [33], Kimi K3 adopts auxiliary-loss-free routing [30].”

Quantile Balancing core

“we introduce Quantile Balancing (QB), which sets each expert bias from the router-score quantile that matches its target load [111].”

Exact coordinate minimization used

“each subproblem admits a closed-form exact solution.”

Exponential moving average of estimated quantiles used

“maintaining an exponential moving average of the estimated quantiles across steps reduces batch-to-batch sampling noise and can improve…”

Histogram-based global quantile estimation used

“We instead read each expert’s quantile from a histogram of its margins: a single all-reduce sums the per-rank bin counts, and the quantile…”

Pooled-global-batch quantile estimation used

“estimate is the quantile of the pooled global batch rather than an average of per-rank quantiles, which generally differs.”

Uniform-bin histogram quantile estimation used

“At the end of the step, a single all-reduce sums the local counts into the global histogram, and every rank recovers the quantile from the…”

latent mixture of experts 3

LatentMoE core

“LatentMoE [32] makes this expansion affordable by separating the full model width from the routed-expert width”

Stable LatentMoE core

“Stable LatentMoE addresses these two failure modes with three components: an RMSNorm before the up-projection and Sigmoid Tanh Unit GLU…”

Sharded latent weights with fused all-gather GEMM epilogue core

“we shard latent weight matrices across ranks and fuse the output all-gather into the GEMM epilogue using multimem store instructions.”

positional encoding 1

No Position Encoding core

“Kimi K3 uses no explicit positional embedding (NoPE), and instead encodes positional information implicitly through the recurrent gating…”

normalization & residual 3

Attention Residuals core

“Attention Residuals (AttnRes) [57] allows each layer to selectively attend to representations from all preceding layers.”

Block Attention Residuals core

“For the attention residual, we design a companion optimization based on Block AttnRes.”

Two-phase Block AttnRes schedule core

“Block AttnRes [57] follows a two-phase schedule: a batched inter-block pass reads the cached block representations once per block, after…”

multimodal architecture 7

Native multimodal architecture core

“text, images, and videos are processed by a single shared backbone within one context, with no post-hoc modality-alignment stage.”

Multimodal input used

“Since K3 accepts image input”

Decoupled Encoder Process (DEP) used

“we introduced the Decoupled Encoder Process (DEP) [59], which splits ViT and text training into separate stages and balances vision…”

Dual-format coordinate supervision used

“coordinate supervision is provided in both absolute and normalized ([0,1]) formats, enabling precise and resolution-robust localization.”

From-scratch MoonViT-V2 training with next-token prediction used

“we train Kimi K3 vision encoder, MoonViT-V2, entirely from scratch with next-token prediction.”

MoonViT-V2 used

“Vision Encoder: MoonViT-V2”

PixelShuffle 2×2 visual-token downsampling used

“a pixel-shuffle operation with 2×2 downsampling reduces the number of visual tokens by a factor of four”

context capacity 1

Progressive context extension used

“We achieve this through extending the context window progressively as training proceeds, following a four-stage curriculum. The window…”

1 more mentioned or evaluated
Full Attention Residuals mentioned

normalization & residual “For each layer l, we define a layer-specific learnable pseudo-query ql =w l ∈R d and keys and values”

filed at the root 1

Hyperparameter scaling-law fitting used

“we conduct dedicated scaling-law studies to retune key hyperparameters, including the batch size, learning rate, tokens-per-parameter…”

optimizer 3

Muon used

“Following Kimi K2, Kimi K3 adopts Muon [53] as the optimizer for its matrix parameters.”

Per-Head Muon used

“We optimize the model using the Per-Head Muon optimizer (§ 2.5) together with the weight-clipping mechanism introduced in Kimi K2”

Peer-to-peer shard retrieval for Muon orthogonalization used

“each rank retrieves only the shards of its locally owned parameters via peer-to-peer (P2P) communication with the corresponding owner ranks”

learning-rate schedule 1

Cosine decay default

“Our scaling-law study consistently favors cosine decay over Warmup Stable Decay (WSD), leading us to adopt cosine decay as the default…”

training precision 2

Block-wise FP8 activation quantization with offload used

“In Kimi K3, most activations use block-wise FP8 quantization [58, 30] combined with offload/remote-offload”

FP32 attention-output retention used

“we adopt the method of [98] and keep the attention output in FP32 during training.”

quantization-aware training 3

Quantization-Aware Training used

“In addition, we apply quantization-aware training (QAT) from the SFT stage onward, with MXFP4 weights and MXFP8 activations (§ 4.1.4).”

MXFP4 Quantization-Aware Training used

“To reduce memory footprint and serving cost at deployment, we quantize the MoE expert weights — which dominate the model’s parameter…”

MXFP4 Weights with MXFP8 Activations used

“Kimi K3 applies quantization-aware training from the SFT stage onward, using MXFP4 weights with MXFP8 activations for broad hardware…”

training stability 3

Per-token regularization for off-policy stability used

“Our policy optimization algorithm inherently tolerates such an extreme off-policy regime through a per-token regularization. By…”

Soft dropping used

“soft dropping for overflow tokens”

Weight clipping used

“We optimize the model using the Per-Head Muon optimizer (§ 2.5) together with the weight-clipping mechanism introduced in Kimi K2”

training parallelism 12

MoonEP core

“For a router output I, the planning objective is to minimize the maximum number of redundant experts on any rank”

Redundant-expert capacity reservation core

“Reserving E/R redundant-expert slots per rank therefore guarantees that planning always admits a feasible solution, so training is never…”

Sequence parallelism for activations core

“We therefore adopt sequence parallelism (SP) for activations: the TP all-reduce is decomposed into a reduce-scatter and an all-gather…”

Expert parallelism used

“keeping expert-parallel traffic inside one high-bandwidth domain”

Cache-based pipeline communication used

“For pipeline parallelism, we adopt cache-based pipeline communication [57], in which only newly generated blocks are incrementally…”

Dynamic context parallelism for multimodal samples used

“we extend context parallelism to such large samples. A single large image is partitioned along the patch dimension across multiple…”

Fully balanced expert-parallel training used

“we introduce a fully balanced expert-parallel training method with static shapes and no host synchronization on the critical path.”

GPU planning kernel for redundant expert migration used

“We therefore compute exact solutions offline with integer linear programming (ILP) for representative cases as references and design a GPU…”

Intra-device context parallelism used

“An automatic SM-level context-parallel (CP) planner therefore partitions the sequence across the SMs of a single rank, evaluates the…”

KDA Context Parallelism used

“To preserve this dependence, we introduce KDA Context Parallelism (KCP), which decomposes the effect of each segment into two locally…”

Pipeline ZeRO-2 gradient sharding with CPU offloading used

“we use Pipeline ZeRO-2 gradient sharding [145] to shard gradients across data-parallel (DP) ranks. Furthermore, we store the sharded…”

pipeline-bubble scheduling of ViT computation used

“the remaining forward passes are scheduled into pipeline bubbles, and the backward passes are handled analogously.”

training runtime 7

Composable tensor-granularity activation storage policies core

“Recomputation, quantization, and offload/remote-offload are merely storage policies under this abstraction and can be freely composed at…”

Unified pluggable activation storage manager core

“We design a unified storage abstraction for activations, in which every tensor saved for the backward pass is associated with a pluggable…”

Cross-rank remote activation offloading used

“we remotely offload activations to the memory of other PP ranks using the Mooncake Transfer Engine [96], achieving balanced activation…”

Double-buffered chunked streaming of reference-model weights used

“We stream reference weights into these slots chunk by chunk: one slot is used for the current forward computation while the other…”

Element-wise activation recomputation used

“element-wise operators are configured with recomputation.”

Policy-model gradient-buffer reuse for reference-model weights used

“We keep these weights in CPU memory and materialize them only when needed, backing their parameter tensors with the policy model’s FP32…”

Post-iteration NVMe offloading of training states used

“we offload training states (model weights and optimizer states) to NVMe after a training iteration finishes.”

Not used 1

Warmup-Stable-Decay not used

learning-rate schedule “Our scaling-law study consistently favors cosine decay over Warmup Stable Decay (WSD), leading us to adopt cosine decay as the default…”

data sourcing 1

Agent-driven recursive knowledge graph construction used

“We construct the knowledge graph as a directed acyclic graph through recursive, agent-driven expansion. The expansion process begins with…”

data filtering 2

Domain filtering with heuristics, classifier scoring, and deduplication used

“Each domain is filtered by a combination of rule-based heuristics, classifier-based quality scoring, and deduplication, with…”

Long-context data cleaning and filtering pipeline used

“We therefore process them through a dedicated cleaning pipeline that combines exact and fuzzy deduplication, supplemented by perceptual…”

synthetic data 7

Agentic visual reasoning with a Python interpreter in a sandbox used

“Each visual-reasoning trajectory is generated in an agent environment equipped with a Python interpreter in an isolated sandbox: the model…”

Knowledge and mathematics corpus rephrasing with fidelity verification used

“we rephrase knowledge and mathematics corpora with style and perspective-diverse prompting, chunk-wise autoregressive generation, and…”

Knowledge-graph-guided task synthesis used

“To control both granularity and coverage at scale, we build a self-evolving, hierarchically organized knowledge graph that agents…”

Multi-stage verification with human-in-the-loop annotation used

“we synthesize data trajectories using domain-specialized models from the prior Kimi series, followed by multi-stage verification and…”

Programmatic multimodal data coupling code with rendered visuals used

“we substantially scale up programmatic multimodal data, coupling code snippets with their rendered visuals across domain-specific formats…”

Synthetic long-context data via permutation and concatenation of multimodal documents used

“To address this, we synthesize additional long-context data by carefully permuting and concatenating multimodal documents and sub-tasks…”

Synthetic trajectory generation with domain-specialized models used

“Specifically, we synthesize data trajectories using domain-specialized models from the prior Kimi series, followed by multi-stage…”

data mixture & curriculum 1

Upsampling long-context data used

“Because genuinely long and coherent documents and videos are scarce relative to short text, we upsample them so that the long-context…”

tokenization 1

XTML core

“the template adopts XTML (eXtensible Token Markup Language), an XML-like markup in which the angle-bracket syntax is replaced by three…”

filed at the root 1

Three-stage post-training with SFT, RL, and multi-teacher on-policy distillation core

“Our post-training pipeline follows a three-stage paradigm: initializing baseline agent capabilities via supervised fine-tuning (SFT)…”

supervised fine-tuning 1

Supervised fine-tuning used

“The SFT stage establishes a high-quality cold-start policy for the subsequent RL stage.”

reinforcement learning algorithm 2

Multi-domain, multi-effort reinforcement learning core

“Kimi K3 undergoes reinforcement learning across long-horizon coding, general agents, general reasoning and knowledge tasks, each spanning…”

Domain-specialized RL experts used

“Rather than training specialized RL models for individual tasks, we scale RL across three broad domains, each encompassing a wide spectrum…”

reward modelling 3

Agentic Generative Reward Model used

“For non-verifiable general tasks, we adopt an Agentic Generative Reward Model (GRM), retaining the tournament-style group reward with…”

Public and hidden verifier pairing used

“Reward hacking is mitigated by isolating agents from verifiers, pairing public verifiers that offer diagnostic feedback with hidden…”

Web-development reward with deterministic checks and model judging used

“Rewards consist of two components: deterministic checks and model judging by an internal reward model. Deterministic checks functionally…”

preference optimization 1

Budget-based verbosity control used

“To mitigate reward hacking toward increasingly verbose outputs, we apply a budget-based verbosity control analogous to the…”

policy distillation 2

Multi-Teacher On-Policy Distillation core

“We adopt Multi-Teacher On-Policy Distillation (MOPD) to consolidate these domain-specialized capabilities across varying reasoning efforts…”

Per-Token On-Policy Distillation Reward with Clipping used

“the per-token OPD reward evaluated on yt between the teacher π(d,e)teacher and the student πθ is defined as: rdopd(yt |e, x, y<t) =…”

rollout & RL infrastructure 3

Partial rollout used

“To mitigate the long-tail latency that intensifies in long-horizon tasks, we extend the partial rollout scheme from our synchronous RL…”

Co-located rollout and training used

“We adopt co-located RL training [58] to keep each 1M-context Kimi K3 RL experiment within a few hundred GPUs”

Co-located million-token agentic RL system with resumable sandboxes used

“our co-located system combines partial rollouts, external KV-cache retention, adaptive throttling and resumable microVM sandboxes to…”

agentic post-training 4

Training on verifiable problems in agentic environments core

“We train Kimi K3 on verifiable problems in agentic environments; representative examples include multi-step complex information searching…”

Autonomous Execution Tasks used

“We introduce Autonomous Execution Tasks (AET), an environment paradigm that trains long-horizon agent intelligence through…”

Mock applications for personal-assistant RL used

“For long-horizon personal assistant tasks, we develop realistic mock implementations of widely used applications, such as Gmail, Notion…”

Unified white-box RL environment used

“To address this, we develop a unified white-box RL environment that represents an agent harness as a collection of configurable…”

decoding strategy 2

Fused recurrent replay kernel core

“The replayed tokens, the bonus token, and the next draft window share one recurrent loop inside a single fused kernel”

MTP-to-EAGLE-3 draft-model fine-tuning used

“As the draft model of EAGLE-3 comprises a single decoder layer whose structure matches the MTP layer, we fine-tune the pre-trained MTP…”

reasoning control 8

Always-on thinking mode core

“Kimi K3 always has thinking enabled, and will return `reasoning_content`.”

Thinking mode default

“At launch, Kimi K3 will use max thinking effort by default, with low- and high-effort modes to be introduced in subsequent updates.”

Chain-of-thought reasoning default

“It only has one reasoning effort right now, 'max'—and it shows. The model consumed 13,241 reasoning tokens to output 3,417 tokens of…”

configurable reasoning effort via reasoning_effort (low/high/max) default

“Thinking effort is configured with the top-level reasoning_effort request field, which supports "low", "high", and "max" (default "max").”

Reasoning token-budget control used

“To fine-tune reasoning effort while maximizing token efficiency, we implement a per-problem budget control mechanism during RL”

Generation-prefix mode selection used

“The two generation modes are selected purely through the generation prefix—[open]think[sep] for thinking mode and [open]response[sep] for…”

Natural-language reasoning-effort option message used

“Reasoning effort is exposed as a global option message of type thinking-effort, inserted after the tool declaration and before the input…”

Stage-wise effort-budget curriculum used

“Training follows a stage-wise curriculum over the budget multiplier τ. We first train a max-budget variant with a relatively large τ…”

KV cache management 9

Fine-grained prefix hashing within cache pages core

“Prefix hashing runs on fine hash blocks (e.g., 512 tokens) inside MLA pages, while the physical block remains the coarse allocation unit.”

KDA-aware prefix-cache management core

“a KDA-aware prefix cache packs the fixed-size recurrent state into the same paged pool as the MLA KV cache and keeps long prefixes…”

Projected-input caching for speculative KDA rollback core

“We therefore cache only these projected inputs, rebuild the states of accepted tokens on-chip, and write back the states of the verified…”

Shared-free-list cache allocation core

“First, all cache groups draw blocks from one shared free list, so allocating a private copy for one group could evict a block that another…”

Sparse hash-aligned KDA recurrent-state checkpoints core

“checkpoints of the recurrent state are saved only at (a sparse subset of) MLA’s hash endpoints—the only positions a lookup can ever…”

Unified paged cache layout for KDA states and MLA KV core

“We therefore pack KDA states into the same paged block pool as MLA KV , unifying pages to the same byte size so that both page types share…”

Write-back external KV-cache policy core

“We therefore decouple prefix retention from GPU residency with a write-back design.”

KDA with prefill cache used

“KDA with prefill cache allows us to serve Kimi K3 at a highly competitive token price despite its scale and long context.”

KV-cache-aware placement of one-shot option messages used

“One-shot options( tool_choice, response_format) are appended after the input messages, so that per-request changes leave the history KV…”

inference quantization 3

Offline weight-layout permutation core

“the weight layout is permuted offline at a one-time preprocessing cost, substantially reducing the runtime dequantization overhead.”

MXFP4 weight quantization used

“MXFP4 weights (Microscaling FP4): Each weight is stored in 4-bit floating point with per-block scaling factors.”

MXFP8 activation quantization used

“MXFP8 activations: Intermediate activations use 8-bit floating point, providing higher precision where it matters most”

serving parallelism 1

Zero-copy fused token permutation and unpermutation core

“We implement a fused per-mute/unpermute operator in which the planning kernel precomputes the destination of every token, so tokens are…”

inference scheduling 6

Cross-group cache-hit block pinning core

“every hit block is therefore pinned across all groups before anything is allocated.”

Dual-cluster consistent-hash failover core

“Consistent hashing therefore pins each session to two clusters, a primary that serves its traffic and a pre-assigned secondary that takes…”

Prefix-cache-aware session affinity scheduling core

“cache-aware affinity scheduling routes each session to the cluster holding its prefix cache while bounding the cost of cluster failures”

Request-class resource-budget admission control core

“We therefore adopt budget-based admission control, allocating separate resource budgets to different request classes so that bursty…”

Runtime-signal-based rollout concurrency auto-throttling core

“We therefore design an auto-throttling mechanism at the LLM request scheduling layer, using runtime signals such as active request count…”

Workload-aware routed-expert GEMM scheduling core

“We therefore schedule the routed-expert GEMM with a workload-aware scheduler that adapts its parameters to the current token distribution…”

inference kernel 9

Fused AttnRes merge and RMSNorm core

“the merging of the AttnRes output with its partial-sum update, together with the subsequent RMSNorm, is fused into the preceding TP…”

Fused latent down-projection and MoE-router GEMM core

“First, we fuse the latent down-projection with the MoE router into a single GEMM.”

GPU kernel for online expert placement planning core

“We therefore compute exact solutions offline with integer linear programming (ILP) for representative cases as references and design a GPU…”

Separate-stream shared-expert GEMM overlap core

“For the shared experts, we dispatch their GEMMs to a separate stream so that they overlap with other kernels.”

Side-stream overlap for inter-block attention-residual computation core

“For decoding, we launch the inter-block kernel on a side stream so that it overlaps with independent computation on the main stream.”

Synchronization-free static-shape MoE execution core

“This eliminates the per-layer MoE host synchronization and alleviates the host-side kernel-launch overhead.”

WarpDecode token-centric routed-expert decoding core

“We instead build the MoE decoding kernel upon the token-centric design of WarpDecode [12], in which each warp is responsible for one…”

FlashKDA used

“We therefore develop FlashKDA, a CUTLASS-based chunkwise kernel that overlaps intra-chunk computation with cross-chunk state propagation.…”

KDA algorithm-system co-design used

“For systems co-design for KDA, we develop fused kernels, KDA Context Parallelism, and state-aware prefix caching to make KDA efficient…”

context management 2

Preserved thinking history core

“the think channel is always retained in the history—kept even when its content is empty—so that the model observes a consistent message…”

Context compaction used

“We adopt the context-compaction strategy used in the Claude model cards, triggered at 300K tokens.”

agentic scaffolding 12

Function calling core

“fork creates a new sandbox from the exact state of the original one while keeping the original running, which is useful for reward judging…”

Natural-language instructions for request options core

“tool_choice, response_format, and thinking-effort are each translated into a short natural-language instruction placed in context, rather…”

Periodic sandbox snapshots core

“snapshots of a sandbox can be saved at regular intervals for error recovery.”

Concurrent subagent orchestration used

“Kimi K3 analyzed 391 gravitational-wave events in GWTC-5 using more than 20 concurrent subagents, producing seven scientific…”

Copy-on-write memory with page-cache optimization used

“We further reduce memory usage with copy-on-write memory and page-cache optimizations”

Dynamic tool loading used

“tools retrieved or loaded during a conversation are announced through an additional tool-declare message, after which the model’s…”

Indexed parallel tool calls used

“each call carries tool and index attributes; the index numbers parallel calls within a message, and each tool-result message repeats the…”

microVM-isolated sandbox runtime for agentic AI used

“a new microVM-based sandbox runtime called AgentENV.”

OverlayBD shared-image sandbox launch stack used

“We adopt OverlayBD [68] as the image format, together with a custom ublk driver implementation, storage-layer sharing, and P2P transport…”

Typed tool arguments used

“Arguments are typed: string arguments appear as raw text, while values of other JSON types are compactly serialized. A pure-JSON fallback…”

Vision in the loop used

“Kimi K3 achieves true "vision in the loop" by seamlessly iterating between code and live screenshots—instantly seeing and refining outputs.”

XTML chat template used

“To represent these complex agentic trajectories consistently, we serialize all data with our XTML-based chat template (eXtensible Token…”

2 more mentioned or evaluated
Agentic tool calling mentioned

agentic scaffolding “the thing that matters most for today's model: agentic tool calling and the ability to operate tools reliably as conversations grow in…”

MoonEP token migration for load balancing unclear

serving parallelism “We repeatedly pick an underloaded rank and an overloaded rank, and migrate tokens from the overloaded rank to fill the underloaded rank…”

kernel & quantization library 3

MiniTriton used

“MiniTriton, a Triton-like compiler K3 built from scratch”

MiniTriton 5 used

“Kimi K3 developed MiniTriton 5, a compact Triton-like compiler with a custom tile-level Python frontend and layout system, a lightweight…”

MiniTriton dual-mode tensor library used

“Built around the compiler is a dual-mode tensor library with a PyTorch-like high-level interface, whose eager and forward-only compiled…”

agent product 1

OpenClaw used

“The so-called harness lets coders more easily swap in and out various AI models”

infrastructure service 3

Incremental sandbox checkpointing and resumption core

“AgentENV supports incremental checkpointing and resuming of sandbox states, where only memory pages dirtied since the last checkpoint are…”

Sandbox pause and resume core

“a paused sandbox consumes no memory or CPU resources; a sandbox can therefore be paused while the agent is waiting for the model’s…”

Mooncake used

“Moonshot's own serving uses the Mooncake disaggregated inference infrastructure, which separates prefill and decode across different node…”

1 more mentioned or evaluated
Triton mentioned

kernel & quantization library “charted against Triton on an Nvidia L20”

filed at the root 6

Exploit development evaluation used

“This tier requires the model to convert a vulnerability into a working end-to-end exploit, and is the tier most directly relevant to…”

Prompt-enforced answering used

“For WorldVQA, we observe consistent refusal behavior across models and enforce an answer via prompt engineering.”

Sandboxed GPU-kernel optimization evaluation used

“Each model works independently in an identically configured sandbox, with a budget of up to 24 hours per task for profiling, rewriting…”

Trajectory analysis for failure attribution used

“Trajectory analysis attributes this gap to four recurring failure modes”

Two-tier cybersecurity capability evaluation used

“We evaluate the model’s cybersecurity capability along a two-tier progression of increasing operational risk: vulnerability discovery with…”

Vulnerability discovery evaluation used

“This tier tasks the model with identifying genuine bugs in current codebases—rather than reproducing known vulnerabilities—and…”

benchmark 2

Living in-house benchmark suite core

“These benchmarks are refreshed and expanded frequently, so that they can closely track the model’s evolving failure modes and directly…”

GPU kernel optimization task suite used

“To strengthen Kimi K3’s GPU kernel optimization capabilities, we build a large-scale suite of kernel tasks ranging from single-operator…”

judge 1

Agentic judge protocol used

“the agentic judge is required to follow a mandatory protocol: (1) read the outcome, product, or text output; (2) generate a rubric; (3)…”

human & real-world evaluation 1

Blind expert judging used

“The comparison is performed under blind expert judging, where experts score each output on code quality, feature completeness, visual…”

filed at the root 2

Reward-hacking detection system used

“To ensure that rewards reflect genuine optimization, we develop a hacking-detection system that penalizes reward-hacking strategies such…”

Score-versus-cost efficiency comparison used

“we examine inference cost efficiency by comparing score against per-task cost across four suites covering coding and agentic tasks”

unfiled 8

native multimodal training from the start of training core

“Kimi K3 adopts a native multimodal training strategy in which language and vision are jointly optimized from the start of training, rather…”

pre-training scaling core

“K3 demonstrates that pre-training scaling, paired with architectural innovation, can still deliver step-change gains”

architectural innovation used

“pre-training scaling, paired with architectural innovation”

autonomous chip design and verification with open-source EDA tools used

“In a single 48-hour autonomous run with Kimi Code, Kimi K3 built, optimized, and verified the chip using open-source EDA tools with the…”

iterative refinement of a deliverable via web search and terminal queries used

“The model completed more than 120 rounds of iterative refinement, drawing on a corpus of 87 quarterly reports and 99 original PDFs (more…”

likelihood-based acceptance-rate loss (LK loss) used

“we directly optimize the likelihood-based LK loss, the negative logarithm of the acceptance rate itself, LLK = −log Σ_{x∈V} min(p(x), q(x))”

tool augmentation during evaluation used

“each cell reports the scores without and with tool augmentation (general tools for HLE-Full, Python for the vision benchmarks), in that…”

tool-augmented benchmark evaluation (with and without tools) used

“each cell reports the scores without and with tool augmentation (general tools for HLE-Full, Python for the vision benchmarks), in that…”

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

↑