Moonshot AI
Kimi K3
What Kimi K3’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 2.8T / 104B active
- Context
- 1M
- Attention
- linear hybridKimi Delta Attention (KDA) linear layers + MLA layers, Attention Residuals
- Experts
- 896 routed + 2 shared, top-16 (Stable LatentMoE)
- Input
- text, image
- Licence
- Modified MIT (Kimi K3 License)
- Released
- 2026-07-16
Against the consensus recipe
Its documents state 7 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictionnot in its documents
- Grouped-query attentionnot in its documents
Optimization
Post-training
- Supervised fine-tuningused
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationcore
Inference & serving
- Thinking modedefault
- Multi-Token Predictionnot in its documents
- FP8not in its documents
Checked in its code
Its code shows 8 of the 19 architecture features checked — 4 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from the model repository’s own code (modeling_kimi_k3.py, configuration_kimi_k3.py, modeling_kimi_linear.py) and config.json at revision f831ab6681, by anthropic/claude-sonnet-5 on 2026-09-25.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention | not in its code | not stated | The MLA attention module sets num_key_value_heads equal to num_attention_heads (both 96), so there is no query-group sharing of fewer KV… show evidenceThe MLA attention module sets num_key_value_heads equal to num_attention_heads (both 96), so there is no query-group sharing of fewer KV heads, and attention is MLA rather than GQA anyway. text_config.num_attention_heads = 96text_config.num_key_value_heads = 96 |
| Multi-head latent attention code only | in its code | not stated | KimiMLAAttention down-projects hidden states into a shared kv_lora_rank latent (with a decoupled RoPE key dimension qk_rope_head_dim) and… show evidenceKimiMLAAttention down-projects hidden states into a shared kv_lora_rank latent (with a decoupled RoPE key dimension qk_rope_head_dim) and up-projects per head via kv_b_proj, DeepSeek-V2/V3 style. text_config.kv_lora_rank = 512text_config.qk_rope_head_dim = 64text_config.q_lora_rank = 1536 |
| Sliding-window attention | not in its code | not stated | No sliding_window config key or windowed masking code exists; the only attention pattern switch is between full (MLA) attention layers and… show the whole verdictNo sliding_window config key or windowed masking code exists; the only attention pattern switch is between full (MLA) attention layers and linear KDA layers. |
| Interleaved sliding-window and global attention | not in its code | not stated | There is no interleaving of sliding-window and global attention layers; the fixed interleaving present is between full-attention (MLA) and… show the whole verdictThere is no interleaving of sliding-window and global attention layers; the fixed interleaving present is between full-attention (MLA) and linear-attention (KDA) layers, not local-window vs global attention. |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | not in its code | not stated | No indexer-based top-k token selection mechanism (index_topk, index_n_heads, etc.) appears anywhere in the config or code. |
| Linear-attention or state-space layers alongside full attention | in its code | adopted | The text_config's linear_attn_config defines kda_layers (linear-attention KDA layers) interleaved with full_attn_layers (full MLA softmax… show evidenceThe text_config's linear_attn_config defines kda_layers (linear-attention KDA layers) interleaved with full_attn_layers (full MLA softmax attention layers) across the 93 hidden layers. text_config.linear_attn_config.kda_layers = [1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 14, 15, 17, 18, 19, 21, 22, 23, 25, 26, 27,…text_config.linear_attn_config.full_attn_layers = [4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80… |
| Gated DeltaNet layers | in its code | adopted | KimiDeltaAttention implements Kimi Delta Attention (a Gated DeltaNet variant) with a delta-rule recurrent update, decay gate… show evidenceKimiDeltaAttention implements Kimi Delta Attention (a Gated DeltaNet variant) with a delta-rule recurrent update, decay gate (A_log/dt_bias), and short convolutions on q/k/v, applied to the kda_layers. text_config.linear_attn_config.num_heads = 96text_config.linear_attn_config.short_conv_kernel_size = 4 |
| Mamba-2 layers | not in its code | not stated | No Mamba-2/SSD selective state-space block (mamba_num_heads, ssm_state_size) exists in the code or config; the linear-attention layers use… show the whole verdictNo Mamba-2/SSD selective state-space block (mamba_num_heads, ssm_state_size) exists in the code or config; the linear-attention layers use KDA (delta-rule), not Mamba-2. |
| Learnable attention sink | not in its code | not stated | No per-head learned sink logit is added to the attention softmax denominator anywhere in the eager or MLA attention implementations. |
| Gated attention output code only | in its code | not stated | In KimiMLAAttention, when mla_use_output_gate is true, a sigmoid gate computed from hidden_states (g_proj) multiplies the attention output… show evidenceIn KimiMLAAttention, when mla_use_output_gate is true, a sigmoid gate computed from hidden_states (g_proj) multiplies the attention output before the final projection. text_config.mla_use_output_gate = true |
| QK normalization | not in its code | not stated | KimiMLAAttention has no query/key RMSNorm or LayerNorm applied before the attention dot product (only kv_a_layernorm and q_a_layernorm… show the whole verdictKimiMLAAttention has no query/key RMSNorm or LayerNorm applied before the attention dot product (only kv_a_layernorm and q_a_layernorm normalize the compressed latents, not per-head q/k before attention). |
| Partial RoPE code only | in its code | not stated | MLA queries/keys are split into a nope portion (qk_nope_head_dim=128) and a rope portion (qk_rope_head_dim=64), with only the rope portion… show evidenceMLA queries/keys are split into a nope portion (qk_nope_head_dim=128) and a rope portion (qk_rope_head_dim=64), with only the rope portion carrying positional info via the decoupled key, and mla_use_nope confirms this partial scheme is active. text_config.qk_nope_head_dim = 128text_config.qk_rope_head_dim = 64text_config.mla_use_nope = true |
| YaRN RoPE scaling | code unclear | not stated | The config exposes a rope_scaling attribute in KimiLinearConfig defaulting to None and it is not present as a key in the given… show evidenceThe config exposes a rope_scaling attribute in KimiLinearConfig defaulting to None and it is not present as a key in the given text_config, and no rotary_emb / RoPE application code is shown (self.rotary_emb is set to None and never used), so YaRN scaling cannot be confirmed as active. |
| Layers without positional encoding (NoPE) documents disagree | not in its code | adopted | All full-attention (MLA) layers use mla_use_nope=True uniformly, meaning nope is applied to every softmax-attention layer's designated… show evidenceAll full-attention (MLA) layers use mla_use_nope=True uniformly, meaning nope is applied to every softmax-attention layer's designated portion rather than some layers having positional encoding and others not; there is no layer-varying NoPE pattern. text_config.mla_use_nope = true |
| Mixture of experts | in its code | adopted | KimiSparseMoeBlock routes tokens among num_experts=896 expert MLPs choosing num_experts_per_token=16 via a learned KimiMoEGate router… show evidenceKimiSparseMoeBlock routes tokens among num_experts=896 expert MLPs choosing num_experts_per_token=16 via a learned KimiMoEGate router, applied to layers past first_k_dense_replace. text_config.num_experts = 896text_config.num_experts_per_token = 16text_config.first_k_dense_replace = 1 |
| Shared expert code only | in its code | not stated | KimiSparseMoeBlock builds a shared_experts KimiMLP sized by num_shared_experts=2 that is added to every token's output alongside the… show evidenceKimiSparseMoeBlock builds a shared_experts KimiMLP sized by num_shared_experts=2 that is added to every token's output alongside the routed experts. text_config.num_shared_experts = 2 |
| Auxiliary-loss-free load balancing (selection bias) | in its code | adopted | KimiMoEGate adds a learned e_score_correction_bias to the routing scores solely for top-k expert selection (topk_method=noaux_tc)… show evidenceKimiMoEGate adds a learned e_score_correction_bias to the routing scores solely for top-k expert selection (topk_method=noaux_tc), matching DeepSeek-V3's aux-loss-free bias mechanism. text_config.topk_method = "noaux_tc" |
| Multi-token prediction layers | not in its code | not stated | num_nextn_predict_layers is set to 0 in this config, so no multi-token-prediction modules are built even though the config field exists. show evidencetext_config.num_nextn_predict_layers = 0 |
| SwiGLU feed-forward | not in its code | not stated | hidden_act is set to "situ" and the MLP/expert modules use the custom SituAndMul activation (beta*tanh(gate/beta)*sigmoid(gate)*up), not a… show evidencehidden_act is set to "situ" and the MLP/expert modules use the custom SituAndMul activation (beta*tanh(gate/beta)*sigmoid(gate)*up), not a SiLU-gated SwiGLU. text_config.hidden_act = "situ" |
token mixer 10
“attention is factorized into intra-frame spatial and inter-frame temporal passes, and temporal pooling further compresses tokens along the…”
softmax attention 1
multi-head latent attention 1
“MLA caches ct and reconstructs the content keys and values through learned up-projections during attention computation.”
linear attention & state space 7
gated delta network 7
“KDA extends the delta-rule recurrence [105, 138] with a channel-wise forget gate [63].”
“Its architecture uses KDA and Attention Residuals for computational efficiency.”
“A hybrid linear attention mechanism that replaces standard quadratic attention in a subset of layers.”
“Kimi K3 is built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes)”
“similar to the Gated DeltaNet used for Olmo Hybrid”
“Kimi K3 changes KDA’s output gate from the low-rank parameterization used by Kimi Linear [63] to an input-dependent full-rank projection.”
“Kimi K3 instead uses a scaled sigmoid to bound the log-decay from below”
hybrid layer stacking 1
“Kimi K3 uses a layerwise hybrid of linear and global attention, combining KDA [63] with Gated MLA.”
channel mixer 18
dense feed-forward network 2
“we propose Sigmoid Tanh Unit GLU (SiTU-GLU)”
“Sigmoid Tanh Unit (SiTU): Custom activation function replacing standard GeLU/SwiGLU”
mixture of experts 16
“We have also scaled up Mixture of Experts (MoE) sparsity, effectively activating 16 out of 896 experts when paired with a Stable LatentMoE…”
“activates just 16 of its 896 experts per token”
“during the backward pass, the input of the group GEMM is recovered by recomputing dispatch.”
“we rewrite this gradient through a mathematical transformation into a form that depends only on the intermediate activation act_output and…”
expert routing 2
“at deployment, routing is a fixed Top- k selection with a frozen bias, and no quantile”
“Key components include latent-space routing, Quantile Balancing for load management, and soft dropping for overflow tokens.”
expert load balancing 7
“Unlike auxiliary-loss-based routing [33], Kimi K3 adopts auxiliary-loss-free routing [30].”
“we introduce Quantile Balancing (QB), which sets each expert bias from the router-score quantile that matches its target load [111].”
“each subproblem admits a closed-form exact solution.”
“maintaining an exponential moving average of the estimated quantiles across steps reduces batch-to-batch sampling noise and can improve…”
“We instead read each expert’s quantile from a histogram of its margins: a single all-reduce sums the per-rank bin counts, and the quantile…”
“estimate is the quantile of the pooled global batch rather than an average of per-rank quantiles, which generally differs.”
“At the end of the step, a single all-reduce sums the local counts into the global histogram, and every rank recovers the quantile from the…”
latent mixture of experts 3
“LatentMoE [32] makes this expansion affordable by separating the full model width from the routed-expert width”
“Stable LatentMoE addresses these two failure modes with three components: an RMSNorm before the up-projection and Sigmoid Tanh Unit GLU…”
“we shard latent weight matrices across ranks and fuse the output all-gather into the GEMM epilogue using multimem store instructions.”
positional encoding 1
“Kimi K3 uses no explicit positional embedding (NoPE), and instead encodes positional information implicitly through the recurrent gating…”
normalization & residual 3
“Attention Residuals (AttnRes) [57] allows each layer to selectively attend to representations from all preceding layers.”
“For the attention residual, we design a companion optimization based on Block AttnRes.”
“Block AttnRes [57] follows a two-phase schedule: a batched inter-block pass reads the cached block representations once per block, after…”
multimodal architecture 7
“text, images, and videos are processed by a single shared backbone within one context, with no post-hoc modality-alignment stage.”
“Since K3 accepts image input”
“we introduced the Decoupled Encoder Process (DEP) [59], which splits ViT and text training into separate stages and balances vision…”
“coordinate supervision is provided in both absolute and normalized ([0,1]) formats, enabling precise and resolution-robust localization.”
“we train Kimi K3 vision encoder, MoonViT-V2, entirely from scratch with next-token prediction.”
“Vision Encoder: MoonViT-V2”
“a pixel-shuffle operation with 2×2 downsampling reduces the number of visual tokens by a factor of four”
context capacity 1
“We achieve this through extending the context window progressively as training proceeds, following a four-stage curriculum. The window…”
1 more mentioned or evaluated
normalization & residual “For each layer l, we define a layer-specific learnable pseudo-query ql =w l ∈R d and keys and values”
optimization 33
filed at the root 1
“we conduct dedicated scaling-law studies to retune key hyperparameters, including the batch size, learning rate, tokens-per-parameter…”
optimizer 3
“Following Kimi K2, Kimi K3 adopts Muon [53] as the optimizer for its matrix parameters.”
“We optimize the model using the Per-Head Muon optimizer (§ 2.5) together with the weight-clipping mechanism introduced in Kimi K2”
“each rank retrieves only the shards of its locally owned parameters via peer-to-peer (P2P) communication with the corresponding owner ranks”
learning-rate schedule 1
“Our scaling-law study consistently favors cosine decay over Warmup Stable Decay (WSD), leading us to adopt cosine decay as the default…”
training precision 2
“In Kimi K3, most activations use block-wise FP8 quantization [58, 30] combined with offload/remote-offload”
“we adopt the method of [98] and keep the attention output in FP32 during training.”
quantization-aware training 3
“In addition, we apply quantization-aware training (QAT) from the SFT stage onward, with MXFP4 weights and MXFP8 activations (§ 4.1.4).”
“To reduce memory footprint and serving cost at deployment, we quantize the MoE expert weights — which dominate the model’s parameter…”
“Kimi K3 applies quantization-aware training from the SFT stage onward, using MXFP4 weights with MXFP8 activations for broad hardware…”
training stability 3
“Our policy optimization algorithm inherently tolerates such an extreme off-policy regime through a per-token regularization. By…”
“soft dropping for overflow tokens”
“We optimize the model using the Per-Head Muon optimizer (§ 2.5) together with the weight-clipping mechanism introduced in Kimi K2”
training parallelism 12
“For a router output I, the planning objective is to minimize the maximum number of redundant experts on any rank”
“Reserving E/R redundant-expert slots per rank therefore guarantees that planning always admits a feasible solution, so training is never…”
“We therefore adopt sequence parallelism (SP) for activations: the TP all-reduce is decomposed into a reduce-scatter and an all-gather…”
“keeping expert-parallel traffic inside one high-bandwidth domain”
“For pipeline parallelism, we adopt cache-based pipeline communication [57], in which only newly generated blocks are incrementally…”
“we extend context parallelism to such large samples. A single large image is partitioned along the patch dimension across multiple…”
“we introduce a fully balanced expert-parallel training method with static shapes and no host synchronization on the critical path.”
“We therefore compute exact solutions offline with integer linear programming (ILP) for representative cases as references and design a GPU…”
“An automatic SM-level context-parallel (CP) planner therefore partitions the sequence across the SMs of a single rank, evaluates the…”
“To preserve this dependence, we introduce KDA Context Parallelism (KCP), which decomposes the effect of each segment into two locally…”
“we use Pipeline ZeRO-2 gradient sharding [145] to shard gradients across data-parallel (DP) ranks. Furthermore, we store the sharded…”
“the remaining forward passes are scheduled into pipeline bubbles, and the backward passes are handled analogously.”
training runtime 7
“Recomputation, quantization, and offload/remote-offload are merely storage policies under this abstraction and can be freely composed at…”
“We design a unified storage abstraction for activations, in which every tensor saved for the backward pass is associated with a pluggable…”
“we remotely offload activations to the memory of other PP ranks using the Mooncake Transfer Engine [96], achieving balanced activation…”
“We stream reference weights into these slots chunk by chunk: one slot is used for the current forward computation while the other…”
“element-wise operators are configured with recomputation.”
“We keep these weights in CPU memory and materialize them only when needed, backing their parameter tensors with the policy model’s FP32…”
“we offload training states (model weights and optimizer states) to NVMe after a training iteration finishes.”
Not used 1
learning-rate schedule “Our scaling-law study consistently favors cosine decay over Warmup Stable Decay (WSD), leading us to adopt cosine decay as the default…”
data sourcing 1
“We construct the knowledge graph as a directed acyclic graph through recursive, agent-driven expansion. The expansion process begins with…”
data filtering 2
“Each domain is filtered by a combination of rule-based heuristics, classifier-based quality scoring, and deduplication, with…”
“We therefore process them through a dedicated cleaning pipeline that combines exact and fuzzy deduplication, supplemented by perceptual…”
synthetic data 7
“Each visual-reasoning trajectory is generated in an agent environment equipped with a Python interpreter in an isolated sandbox: the model…”
“we rephrase knowledge and mathematics corpora with style and perspective-diverse prompting, chunk-wise autoregressive generation, and…”
“To control both granularity and coverage at scale, we build a self-evolving, hierarchically organized knowledge graph that agents…”
“we synthesize data trajectories using domain-specialized models from the prior Kimi series, followed by multi-stage verification and…”
“we substantially scale up programmatic multimodal data, coupling code snippets with their rendered visuals across domain-specific formats…”
“To address this, we synthesize additional long-context data by carefully permuting and concatenating multimodal documents and sub-tasks…”
“Specifically, we synthesize data trajectories using domain-specialized models from the prior Kimi series, followed by multi-stage…”
data mixture & curriculum 1
“Because genuinely long and coherent documents and videos are scarce relative to short text, we upsample them so that the long-context…”
tokenization 1
“the template adopts XTML (eXtensible Token Markup Language), an XML-like markup in which the angle-bracket syntax is replaced by three…”
filed at the root 1
“Our post-training pipeline follows a three-stage paradigm: initializing baseline agent capabilities via supervised fine-tuning (SFT)…”
supervised fine-tuning 1
“The SFT stage establishes a high-quality cold-start policy for the subsequent RL stage.”
reinforcement learning algorithm 2
“Kimi K3 undergoes reinforcement learning across long-horizon coding, general agents, general reasoning and knowledge tasks, each spanning…”
“Rather than training specialized RL models for individual tasks, we scale RL across three broad domains, each encompassing a wide spectrum…”
reward modelling 3
“For non-verifiable general tasks, we adopt an Agentic Generative Reward Model (GRM), retaining the tournament-style group reward with…”
“Reward hacking is mitigated by isolating agents from verifiers, pairing public verifiers that offer diagnostic feedback with hidden…”
“Rewards consist of two components: deterministic checks and model judging by an internal reward model. Deterministic checks functionally…”
preference optimization 1
“To mitigate reward hacking toward increasingly verbose outputs, we apply a budget-based verbosity control analogous to the…”
policy distillation 2
“We adopt Multi-Teacher On-Policy Distillation (MOPD) to consolidate these domain-specialized capabilities across varying reasoning efforts…”
“the per-token OPD reward evaluated on yt between the teacher π(d,e)teacher and the student πθ is defined as: rdopd(yt |e, x, y<t) =…”
rollout & RL infrastructure 3
“To mitigate the long-tail latency that intensifies in long-horizon tasks, we extend the partial rollout scheme from our synchronous RL…”
“We adopt co-located RL training [58] to keep each 1M-context Kimi K3 RL experiment within a few hundred GPUs”
“our co-located system combines partial rollouts, external KV-cache retention, adaptive throttling and resumable microVM sandboxes to…”
agentic post-training 4
“We train Kimi K3 on verifiable problems in agentic environments; representative examples include multi-step complex information searching…”
“We introduce Autonomous Execution Tasks (AET), an environment paradigm that trains long-horizon agent intelligence through…”
“For long-horizon personal assistant tasks, we develop realistic mock implementations of widely used applications, such as Gmail, Notion…”
“To address this, we develop a unified white-box RL environment that represents an agent harness as a collection of configurable…”
decoding strategy 2
“The replayed tokens, the bonus token, and the next draft window share one recurrent loop inside a single fused kernel”
“As the draft model of EAGLE-3 comprises a single decoder layer whose structure matches the MTP layer, we fine-tune the pre-trained MTP…”
reasoning control 8
“Kimi K3 always has thinking enabled, and will return `reasoning_content`.”
“At launch, Kimi K3 will use max thinking effort by default, with low- and high-effort modes to be introduced in subsequent updates.”
“It only has one reasoning effort right now, 'max'—and it shows. The model consumed 13,241 reasoning tokens to output 3,417 tokens of…”
“Thinking effort is configured with the top-level reasoning_effort request field, which supports "low", "high", and "max" (default "max").”
“To fine-tune reasoning effort while maximizing token efficiency, we implement a per-problem budget control mechanism during RL”
“The two generation modes are selected purely through the generation prefix—[open]think[sep] for thinking mode and [open]response[sep] for…”
“Reasoning effort is exposed as a global option message of type thinking-effort, inserted after the tool declaration and before the input…”
“Training follows a stage-wise curriculum over the budget multiplier τ. We first train a max-budget variant with a relatively large τ…”
KV cache management 9
“Prefix hashing runs on fine hash blocks (e.g., 512 tokens) inside MLA pages, while the physical block remains the coarse allocation unit.”
“a KDA-aware prefix cache packs the fixed-size recurrent state into the same paged pool as the MLA KV cache and keeps long prefixes…”
“We therefore cache only these projected inputs, rebuild the states of accepted tokens on-chip, and write back the states of the verified…”
“First, all cache groups draw blocks from one shared free list, so allocating a private copy for one group could evict a block that another…”
“checkpoints of the recurrent state are saved only at (a sparse subset of) MLA’s hash endpoints—the only positions a lookup can ever…”
“We therefore pack KDA states into the same paged block pool as MLA KV , unifying pages to the same byte size so that both page types share…”
“We therefore decouple prefix retention from GPU residency with a write-back design.”
“KDA with prefill cache allows us to serve Kimi K3 at a highly competitive token price despite its scale and long context.”
“One-shot options( tool_choice, response_format) are appended after the input messages, so that per-request changes leave the history KV…”
inference quantization 3
“the weight layout is permuted offline at a one-time preprocessing cost, substantially reducing the runtime dequantization overhead.”
“MXFP4 weights (Microscaling FP4): Each weight is stored in 4-bit floating point with per-block scaling factors.”
“MXFP8 activations: Intermediate activations use 8-bit floating point, providing higher precision where it matters most”
serving parallelism 1
“We implement a fused per-mute/unpermute operator in which the planning kernel precomputes the destination of every token, so tokens are…”
inference scheduling 6
“every hit block is therefore pinned across all groups before anything is allocated.”
“Consistent hashing therefore pins each session to two clusters, a primary that serves its traffic and a pre-assigned secondary that takes…”
“cache-aware affinity scheduling routes each session to the cluster holding its prefix cache while bounding the cost of cluster failures”
“We therefore adopt budget-based admission control, allocating separate resource budgets to different request classes so that bursty…”
“We therefore design an auto-throttling mechanism at the LLM request scheduling layer, using runtime signals such as active request count…”
“We therefore schedule the routed-expert GEMM with a workload-aware scheduler that adapts its parameters to the current token distribution…”
inference kernel 9
“the merging of the AttnRes output with its partial-sum update, together with the subsequent RMSNorm, is fused into the preceding TP…”
“First, we fuse the latent down-projection with the MoE router into a single GEMM.”
“We therefore compute exact solutions offline with integer linear programming (ILP) for representative cases as references and design a GPU…”
“For the shared experts, we dispatch their GEMMs to a separate stream so that they overlap with other kernels.”
“For decoding, we launch the inter-block kernel on a side stream so that it overlaps with independent computation on the main stream.”
“This eliminates the per-layer MoE host synchronization and alleviates the host-side kernel-launch overhead.”
“We instead build the MoE decoding kernel upon the token-centric design of WarpDecode [12], in which each warp is responsible for one…”
“We therefore develop FlashKDA, a CUTLASS-based chunkwise kernel that overlaps intra-chunk computation with cross-chunk state propagation.…”
“For systems co-design for KDA, we develop fused kernels, KDA Context Parallelism, and state-aware prefix caching to make KDA efficient…”
context management 2
“the think channel is always retained in the history—kept even when its content is empty—so that the model observes a consistent message…”
“We adopt the context-compaction strategy used in the Claude model cards, triggered at 300K tokens.”
agentic scaffolding 12
“fork creates a new sandbox from the exact state of the original one while keeping the original running, which is useful for reward judging…”
“tool_choice, response_format, and thinking-effort are each translated into a short natural-language instruction placed in context, rather…”
“snapshots of a sandbox can be saved at regular intervals for error recovery.”
“Kimi K3 analyzed 391 gravitational-wave events in GWTC-5 using more than 20 concurrent subagents, producing seven scientific…”
“We further reduce memory usage with copy-on-write memory and page-cache optimizations”
“tools retrieved or loaded during a conversation are announced through an additional tool-declare message, after which the model’s…”
“each call carries tool and index attributes; the index numbers parallel calls within a message, and each tool-result message repeats the…”
“a new microVM-based sandbox runtime called AgentENV.”
“We adopt OverlayBD [68] as the image format, together with a custom ublk driver implementation, storage-layer sharing, and P2P transport…”
“Arguments are typed: string arguments appear as raw text, while values of other JSON types are compactly serialized. A pure-JSON fallback…”
“Kimi K3 achieves true "vision in the loop" by seamlessly iterating between code and live screenshots—instantly seeing and refining outputs.”
“To represent these complex agentic trajectories consistently, we serialize all data with our XTML-based chat template (eXtensible Token…”
2 more mentioned or evaluated
agentic scaffolding “the thing that matters most for today's model: agentic tool calling and the ability to operate tools reliably as conversations grow in…”
serving parallelism “We repeatedly pick an underloaded rank and an overloaded rank, and migrate tokens from the overloaded rank to fill the underloaded rank…”
kernel & quantization library 3
“MiniTriton, a Triton-like compiler K3 built from scratch”
“Kimi K3 developed MiniTriton 5, a compact Triton-like compiler with a custom tile-level Python frontend and layout system, a lightweight…”
“Built around the compiler is a dual-mode tensor library with a PyTorch-like high-level interface, whose eager and forward-only compiled…”
agent product 1
“The so-called harness lets coders more easily swap in and out various AI models”
infrastructure service 3
“AgentENV supports incremental checkpointing and resuming of sandbox states, where only memory pages dirtied since the last checkpoint are…”
“a paused sandbox consumes no memory or CPU resources; a sandbox can therefore be paused while the agent is waiting for the model’s…”
“Moonshot's own serving uses the Mooncake disaggregated inference infrastructure, which separates prefill and decode across different node…”
1 more mentioned or evaluated
kernel & quantization library “charted against Triton on an Nvidia L20”
evaluation 10
filed at the root 6
“This tier requires the model to convert a vulnerability into a working end-to-end exploit, and is the tier most directly relevant to…”
“For WorldVQA, we observe consistent refusal behavior across models and enforce an answer via prompt engineering.”
“Each model works independently in an identically configured sandbox, with a budget of up to 24 hours per task for profiling, rewriting…”
“Trajectory analysis attributes this gap to four recurring failure modes”
“We evaluate the model’s cybersecurity capability along a two-tier progression of increasing operational risk: vulnerability discovery with…”
“This tier tasks the model with identifying genuine bugs in current codebases—rather than reproducing known vulnerabilities—and…”
benchmark 2
“These benchmarks are refreshed and expanded frequently, so that they can closely track the model’s evolving failure modes and directly…”
“To strengthen Kimi K3’s GPU kernel optimization capabilities, we build a large-scale suite of kernel tasks ranging from single-operator…”
judge 1
“the agentic judge is required to follow a mandatory protocol: (1) read the outcome, product, or text output; (2) generate a rubric; (3)…”
human & real-world evaluation 1
“The comparison is performed under blind expert judging, where experts score each output on code quality, feature completeness, visual…”
other 2
filed at the root 2
“To ensure that rewards reflect genuine optimization, we develop a hacking-detection system that penalizes reward-hacking strategies such…”
“we examine inference cost efficiency by comparing score against per-task cost across four suites covering coding and agentic tasks”
unfiled 8
“Kimi K3 adopts a native multimodal training strategy in which language and vision are jointly optimized from the start of training, rather…”
“K3 demonstrates that pre-training scaling, paired with architectural innovation, can still deliver step-change gains”
“pre-training scaling, paired with architectural innovation”
“In a single 48-hour autonomous run with Kimi Code, Kimi K3 built, optimized, and verified the chip using open-source EDA tools with the…”
“The model completed more than 120 rounds of iterative refinement, drawing on a corpus of 87 quarterly reports and 99 original PDFs (more…”
“we directly optimize the likelihood-based LK loss, the negative logarithm of the acceptance rate itself, LLK = −log Σ_{x∈V} min(p(x), q(x))”
“each cell reports the scores without and with tool augmentation (general tools for HLE-Full, Python for the vision benchmarks), in that…”
“each cell reports the scores without and with tool augmentation (general tools for HLE-Full, Python for the vision benchmarks), in that…”
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Kimi K3 architecture card — LLM Architecture Gallery explainer sebastianraschka.com
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.