StepFun · usage rank #5
Step-5-Preview
What Step-5-Preview’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 600B / 27B active
- Context
- 1M
- Attention
- sliding/global hybrid3 sliding-window (512) : 1 full GQA over 92 layers; DSA-style indexer (top-k 512) on full layers
- Experts
- 352 routed + 1 shared, top-8
- Input
- text, image, video
- Licence
- StepFun Model License 1.0
- Released
- 2026-10-08
Rank by open-weight tokens on OpenRouter, week of 2026-10-26.
Against the consensus recipe
Its documents state 4 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictionnot in its documents
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismoptional
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modenot in its documents
- Multi-Token Predictionoptional
- FP8optional
Checked in its code
Its code shows 12 of the 19 architecture features checked — 11 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from vLLM (__init__.py, model.py, config.py, layernorm.py, rms_norm.py, add.py, attention_gate.py, sparse_attention.py, summary_update.py, sparse_prefill.py, sparse_prefill_pages.py, sparse_indexer.py, flashinfer_sparse.py, vl.py) and config.json at revision 364b3b03fe, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention code only | in its code | not stated | 64 query heads share 4 KV groups (num_attention_groups=4) in both full and sliding attention layers via QKVParallelLinear with… show evidence64 query heads share 4 KV groups (num_attention_groups=4) in both full and sliding attention layers via QKVParallelLinear with total_num_kv_heads < total_num_heads. text_config.num_attention_heads = 64text_config.num_attention_groups = 4 |
| Multi-head latent attention | not in its code | not stated | No kv_lora_rank or q_lora_rank config, and the attention implementation uses standard QKVParallelLinear with GQA (att_impl_type GQA), not… show the whole verdictNo kv_lora_rank or q_lora_rank config, and the attention implementation uses standard QKVParallelLinear with GQA (att_impl_type GQA), not a latent low-rank KV cache. |
| Sliding-window attention code only | in its code | not stated | layer_types alternates and sliding_window=512 is set, enabling local-window attention on layers marked sliding_attention. show evidencetext_config.sliding_window = 512text_config.layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "full_attenti… |
| Interleaved sliding-window and global attention code only | in its code | not stated | layer_types fixes a repeating pattern of three sliding_attention layers followed by one full_attention layer across the 92-layer stack. show evidencetext_config.layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "full_attenti… |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) code only | in its code | not stated | A sparse indexer (sparse_config, enabled true) scores past regions and attention selects only the top-k=512 regions via select_regions… show evidenceA sparse indexer (sparse_config, enabled true) scores past regions and attention selects only the top-k=512 regions via select_regions, applied to full_attention layers. text_config.sparse_config.enabled = truetext_config.sparse_config.topk = 512text_config.sparse_config.apply_to_layer_types = ["full_attention"] |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | No linear-attention or state-space mixer layers are present; both layer_types values (sliding_attention, full_attention) are softmax… show the whole verdictNo linear-attention or state-space mixer layers are present; both layer_types values (sliding_attention, full_attention) are softmax attention variants. |
| Gated DeltaNet layers | not in its code | not stated | No gated delta rule / Kimi Delta Attention mechanism exists in this codebase; layer_types contains only sliding_attention and… show the whole verdictNo gated delta rule / Kimi Delta Attention mechanism exists in this codebase; layer_types contains only sliding_attention and full_attention. |
| Mamba-2 layers | not in its code | not stated | No Mamba-2 SSD blocks are implemented or referenced; the model is a GQA transformer with sparse attention. |
| Learnable attention sink | not in its code | not stated | No learned per-head sink logit exists; sinks are explicitly rejected/unsupported in the DSA attention implementation. show evidence |
| Gated attention output code only | in its code | not stated | use_head_wise_attn_gate=true enables a per-head sigmoid gate (g_proj) multiplied into the attention output before the output projection. show evidencetext_config.use_head_wise_attn_gate = true |
| QK normalization code only | in its code | not stated | Queries and keys are each normalized per-head with zero-centered RMSNorm (q_norm/k_norm) before the rotary embedding and attention dot… show evidenceQueries and keys are each normalized per-head with zero-centered RMSNorm (q_norm/k_norm) before the rotary embedding and attention dot product. |
| Partial RoPE code only | in its code | not stated | partial_rotary_factors alternates between 1.0 and 0.333... per layer, so full_attention layers apply RoPE to only one third of the head… show evidencepartial_rotary_factors alternates between 1.0 and 0.333... per layer, so full_attention layers apply RoPE to only one third of the head dimension. text_config.partial_rotary_factors = [1.0, 1.0, 1.0, 0.3333333333333333, 1.0, 1.0, 1.0, 0.3333333333333333, 1.0, 1… |
| YaRN RoPE scaling | not in its code | not stated | rope_scaling.rope_type is 'llama3', not YaRN, so no YaRN scaling is applied despite yarn_only_types gating logic in code. |
| Layers without positional encoding (NoPE) | code unclear | not stated | Code supports a use_rope_layers list that can disable RoPE per layer, but this config does not set use_rope_layers, so whether any… show evidenceCode supports a use_rope_layers list that can disable RoPE per layer, but this config does not set use_rope_layers, so whether any softmax-attention layer runs without RoPE cannot be determined. |
| Mixture of experts | in its code | adopted | Router (GateLinear) selects moe_top_k=8 of moe_num_experts=352 experts per token for layers listed in moe_layer_list. show evidencetext_config.moe_num_experts = 352text_config.moe_top_k = 8text_config.num_experts_per_tok = 8 |
| Shared expert code only | in its code | not stated | Each MoE layer includes a share_expert MLP of intermediate size share_expert_dim=1536 that every token passes through, added to the routed… show evidenceEach MoE layer includes a share_expert MLP of intermediate size share_expert_dim=1536 that every token passes through, added to the routed output. text_config.share_expert_dim = 1536 |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | A per-expert router_bias (e_score_correction_bias) is added only to select top-k experts, while weights come from unbiased sigmoid scores… show evidenceA per-expert router_bias (e_score_correction_bias) is added only to select top-k experts, while weights come from unbiased sigmoid scores, enabled by use_moe_router_bias=true. text_config.use_moe_router_bias = true |
| Multi-token prediction layers code only | in its code | not stated | num_nextn_predict_layers=3 declares three MTP modules, built as Step5MTP and referenced by modules_to_not_convert entries for layers 92-94. show evidencetext_config.num_nextn_predict_layers = 3 |
| SwiGLU feed-forward code only | in its code | not stated | MLP and expert feed-forwards use SiluAndMul (SiLU-gated) activation, explicitly rejecting any activation other than silu. show evidence |
channel mixer 1
mixture of experts 1
“Step 5 Preview is a sparse Mixture-of-Experts (MoE) model with 600B total parameters and 27B active parameters.”
multimodal architecture 1
“Combine images, video, and text to extract and analyze multimodal information.”
context capacity 2
“provides a 1M-token context window”
“Configure Step Plan and enable 1M context.”
training precision 1
“BF16 deployments require a BF16 checkpoint.”
training parallelism 5
“For TEP8 or DEP8, retain --enable-expert-parallel”
“Choose one of the following parallel configurations for a single node with four B300 GPUs:”
“For TEP8 or DEP8, retain `--enable-expert-parallel` and remove `--moe-backend deep_gemm_mega_moe`”
“DEP4: Data and Expert Parallelism”
“DEP16 uses TP=1 and distributes the experts across 16 data-parallel ranks, with eight ranks on each node.”
filed at the root 1
“The standalone implementation converts checkpoints into shards for offline generation.”
decoding strategy 4
“Use a framework build that includes Step 5 Preview support, including the reasoning, tool-call, and speculative decoding features used in…”
“`--enable-multi-layer-eagle`”
“--spec-method mtp \ --spec-tokens 3”
“--speculative-algorithm EAGLE”
KV cache management 1
“`--enable-prefix-caching`”
inference quantization 1
“The examples below cover FP8 and BF16 checkpoints.”
agentic scaffolding 1
“repeated tool calls, execution, verification, and revision”
2 more mentioned or evaluated
serving parallelism “For more optimized configurations, including MegaMoE, attention data parallelism, and HiCache, see the [SGLang Cookbook for Step 5 Preview]”
KV cache management “For more optimized configurations, including MegaMoE, attention data parallelism, and HiCache, see the [SGLang Cookbook for Step 5 Preview]”
filed at the root 1
“The standalone implementation converts checkpoints into shards for offline generation.”
kernel & quantization library 1
“This configuration uses MegaMoE for the FP8 routed experts.”
agent product 1
“It can navigate large codebases, coordinate changes across files, run and verify code, and iterate on the results.”
infrastructure service 1
“we can recover by routing to another healthy provider”
unfiled 3
“Analyze long documents, multiple source materials, and conversation history within a 1M-token context.”
“Use tools provided by an application to retrieve information, process documents, and continue through multi-step work.”
“Step 5 Preview can carry out tasks that require repeated tool calls, execution, verification, and revision”
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| stepfun.com/step-5-preview | official blog | StepFun | not fetched |
| github.com/stepfun-ai/Step-5-Preview | code repo | StepFun | read |
| huggingface.co/stepfun-ai/Step-5-Preview | model card | StepFun | read |
| platform.stepfun.ai/docs/en/guides/models/step-5-preview | vendor docs | StepFun | read |
| openrouter.ai/stepfun/step-5-preview | vendor docs | OpenRouter | read |