Open Models Almanac
ModelsStep-5-Preview

StepFun · usage rank #5

Step-5-Preview

What Step-5-Preview’s own documents say it is built from — every method with the sentence that describes it.

Parameters
600B / 27B active
Context
1M
Attention
sliding/global hybrid3 sliding-window (512) : 1 full GQA over 92 layers; DSA-style indexer (top-k 512) on full layers
Experts
352 routed + 1 shared, top-8
Input
text, image, video
Licence
StepFun Model License 1.0
Released
2026-10-08

Rank by open-weight tokens on OpenRouter, week of 2026-10-26.

Against the consensus recipe

Its documents state 4 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 12 of the 19 architecture features checked — 11 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from vLLM (__init__.py, model.py, config.py, layernorm.py, rms_norm.py, add.py, attention_gate.py, sparse_attention.py, summary_update.py, sparse_prefill.py, sparse_prefill_pages.py, sparse_indexer.py, flashinfer_sparse.py, vl.py) and config.json at revision 364b3b03fe, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention code only in its codenot stated

64 query heads share 4 KV groups (num_attention_groups=4) in both full and sliding attention layers via QKVParallelLinear with…

show evidence

64 query heads share 4 KV groups (num_attention_groups=4) in both full and sliding attention layers via QKVParallelLinear with total_num_kv_heads < total_num_heads.

text_config.num_attention_heads = 64text_config.num_attention_groups = 4
self.num_heads = self.total_num_heads // tp_size
        self.total_num_kv_heads = num_kv_heads
model.py · L923–924
Multi-head latent attention not in its codenot stated

No kv_lora_rank or q_lora_rank config, and the attention implementation uses standard QKVParallelLinear with GQA (att_impl_type GQA), not…

show the whole verdict

No kv_lora_rank or q_lora_rank config, and the attention implementation uses standard QKVParallelLinear with GQA (att_impl_type GQA), not a latent low-rank KV cache.

Sliding-window attention code only in its codenot stated

layer_types alternates and sliding_window=512 is set, enabling local-window attention on layers marked sliding_attention.

show evidencetext_config.sliding_window = 512text_config.layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "full_attenti…
enable_sliding_window = layer_type == "sliding_attention"
        if yarn_only_types and layer_type not in yarn_only_types:
            rope_scaling = None

        if sliding_window is not None and enable_sliding_window:
model.py · L895–899
Interleaved sliding-window and global attention code only in its codenot stated

layer_types fixes a repeating pattern of three sliding_attention layers followed by one full_attention layer across the 92-layer stack.

show evidencetext_config.layer_types = ["sliding_attention", "sliding_attention", "sliding_attention", "full_attenti…
default_layer_type = (
            "sliding_attention" if self.layer_idx % 2 == 0 else "full_attention"
        )
        layer_type = _per_layer_value(
            layer_types,
            self.layer_idx,
            name="layer_types",
            default=default_layer_type,
        )
model.py · L886–894
Indexer-selected sparse attention (DeepSeek Sparse Attention) code only in its codenot stated

A sparse indexer (sparse_config, enabled true) scores past regions and attention selects only the top-k=512 regions via select_regions…

show evidence

A sparse indexer (sparse_config, enabled true) scores past regions and attention selects only the top-k=512 regions via select_regions, applied to full_attention layers.

text_config.sparse_config.enabled = truetext_config.sparse_config.topk = 512text_config.sparse_config.apply_to_layer_types = ["full_attention"]
topk = values["topk"]
    if not 1 <= topk <= _STEP5_SPARSE_TOPK_MAX:
        raise ValueError(f"Step5 DSA requires topk in [1, 1024], got {topk}.")
config.py · L148–150
self.use_dsa_backend = sparse_config is not None and sliding_window is None
model.py · L1008
Linear-attention or state-space layers alongside full attention not in its codenot stated

No linear-attention or state-space mixer layers are present; both layer_types values (sliding_attention, full_attention) are softmax…

show the whole verdict

No linear-attention or state-space mixer layers are present; both layer_types values (sliding_attention, full_attention) are softmax attention variants.

Gated DeltaNet layers not in its codenot stated

No gated delta rule / Kimi Delta Attention mechanism exists in this codebase; layer_types contains only sliding_attention and…

show the whole verdict

No gated delta rule / Kimi Delta Attention mechanism exists in this codebase; layer_types contains only sliding_attention and full_attention.

Mamba-2 layers not in its codenot stated

No Mamba-2 SSD blocks are implemented or referenced; the model is a GQA transformer with sparse attention.

Learnable attention sink not in its codenot stated

No learned per-head sink logit exists; sinks are explicitly rejected/unsupported in the DSA attention implementation.

show evidence
if sinks is not None:
        raise NotImplementedError("Step5 DSA does not support attention sinks.")
sparse_attention.py · L182–183
Gated attention output code only in its codenot stated

use_head_wise_attn_gate=true enables a per-head sigmoid gate (g_proj) multiplied into the attention output before the output projection.

show evidencetext_config.use_head_wise_attn_gate = true
sigmoid = tl.div_rn(1.0, 1.0 + tl.extra.cuda.libdevice.exp(-gate))
attention_gate.py · L35
attn_output = headwise_sigmoid_gate(
                attn_output, extra_dims, slot_mapping=decode_slots
            )
model.py · L1413–1415
QK normalization code only in its codenot stated

Queries and keys are each normalized per-head with zero-centered RMSNorm (q_norm/k_norm) before the rotary embedding and attention dot…

show evidence

Queries and keys are each normalized per-head with zero-centered RMSNorm (q_norm/k_norm) before the rotary embedding and attention dot product.

self.q_norm = make_step5_rms_norm(
            self.head_dim,
            eps=rms_norm_eps,
            zero_centered=self.zero_centered,
            dtype=norm_dtype,
        )
model.py · L1127–1132
Partial RoPE code only in its codenot stated

partial_rotary_factors alternates between 1.0 and 0.333... per layer, so full_attention layers apply RoPE to only one third of the head…

show evidence

partial_rotary_factors alternates between 1.0 and 0.333... per layer, so full_attention layers apply RoPE to only one third of the head dimension.

text_config.partial_rotary_factors = [1.0, 1.0, 1.0, 0.3333333333333333, 1.0, 1.0, 1.0, 0.3333333333333333, 1.0, 1…
self.partial_rotary_factor = float(partial_rotary_factor)
        self.rotary_dim = int(self.head_dim * self.partial_rotary_factor)
model.py · L954–955
YaRN RoPE scaling not in its codenot stated

rope_scaling.rope_type is 'llama3', not YaRN, so no YaRN scaling is applied despite yarn_only_types gating logic in code.

show evidencetext_config.rope_scaling = {"rope_type": "llama3", "factor": 1.0, "original_max_position_embeddings": 10…
Layers without positional encoding (NoPE) code unclearnot stated

Code supports a use_rope_layers list that can disable RoPE per layer, but this config does not set use_rope_layers, so whether any…

show evidence

Code supports a use_rope_layers list that can disable RoPE per layer, but this config does not set use_rope_layers, so whether any softmax-attention layer runs without RoPE cannot be determined.

self.use_rope = bool(
            _per_layer_value(
                use_rope_layers,
                self.layer_idx,
                name="use_rope_layers",
                default=True,
            )
        )
model.py · L1153–1160
Mixture of experts in its codeadopted

Router (GateLinear) selects moe_top_k=8 of moe_num_experts=352 experts per token for layers listed in moe_layer_list.

show evidencetext_config.moe_num_experts = 352text_config.moe_top_k = 8text_config.num_experts_per_tok = 8
self.gate = GateLinear(
            config.hidden_size,
            config.moe_num_experts,
            bias=False,
            out_dtype=torch.float32,
            params_dtype=torch.float32,
            prefix=f"{prefix}.gate",
        )
model.py · L1700–1707
Shared expert code only in its codenot stated

Each MoE layer includes a share_expert MLP of intermediate size share_expert_dim=1536 that every token passes through, added to the routed…

show evidence

Each MoE layer includes a share_expert MLP of intermediate size share_expert_dim=1536 that every token passes through, added to the routed output.

text_config.share_expert_dim = 1536
self.share_expert = Step5MLP(
            config=config,
            hidden_size=self.hidden_size,
            intermediate_size=config.share_expert_dim,
            hidden_act="silu",
model.py · L1753–1757
Auxiliary-loss-free load balancing (selection bias) code only in its codenot stated

A per-expert router_bias (e_score_correction_bias) is added only to select top-k experts, while weights come from unbiased sigmoid scores…

show evidence

A per-expert router_bias (e_score_correction_bias) is added only to select top-k experts, while weights come from unbiased sigmoid scores, enabled by use_moe_router_bias=true.

text_config.use_moe_router_bias = true
self.router_bias = nn.Parameter(
            torch.zeros(config.moe_num_experts, dtype=torch.float32),
            requires_grad=False,
        )
model.py · L1710–1713
e_score_correction_bias=self.router_bias,
            routed_scaling_factor=config.moe_router_scaling_factor,
model.py · L1777–1778
Multi-token prediction layers code only in its codenot stated

num_nextn_predict_layers=3 declares three MTP modules, built as Step5MTP and referenced by modules_to_not_convert entries for layers 92-94.

show evidencetext_config.num_nextn_predict_layers = 3
SwiGLU feed-forward code only in its codenot stated

MLP and expert feed-forwards use SiluAndMul (SiLU-gated) activation, explicitly rejecting any activation other than silu.

show evidence
if hidden_act != "silu":
            raise ValueError(
                f"Unsupported activation: {hidden_act}. Only silu is supported for now."
            )
        self.act_fn = SiluAndMul()
model.py · L812–816

channel mixer 1

mixture of experts 1

Mixture of experts core

“Step 5 Preview is a sparse Mixture-of-Experts (MoE) model with 600B total parameters and 27B active parameters.”

multimodal architecture 1

Multimodal input understanding core

“Combine images, video, and text to extract and analyze multimodal information.”

context capacity 2

1M-token context window core

“provides a 1M-token context window”

Enabling 1M context in Step Plan optional

“Configure Step Plan and enable 1M context.”

training precision 1

BF16 precision optional

“BF16 deployments require a BF16 checkpoint.”

training parallelism 5

Expert parallelism optional

“For TEP8 or DEP8, retain --enable-expert-parallel”

Tensor parallelism optional

“Choose one of the following parallel configurations for a single node with four B300 GPUs:”

Tensor and expert parallelism optional

“For TEP8 or DEP8, retain `--enable-expert-parallel` and remove `--moe-backend deep_gemm_mega_moe`”

Data and expert parallelism optional

“DEP4: Data and Expert Parallelism”

Data parallelism optional

“DEP16 uses TP=1 and distributes the experts across 16 data-parallel ranks, with eight ranks on each node.”

filed at the root 1

Offline generation optional

“The standalone implementation converts checkpoints into shards for offline generation.”

decoding strategy 4

Speculative decoding used

“Use a framework build that includes Step 5 Preview support, including the reasoning, tool-call, and speculative decoding features used in…”

Multi-layer EAGLE used

“`--enable-multi-layer-eagle`”

Multi-Token Prediction optional

“--spec-method mtp \ --spec-tokens 3”

EAGLE optional

“--speculative-algorithm EAGLE”

KV cache management 1

Prefix caching optional

“`--enable-prefix-caching`”

inference quantization 1

FP8 optional

“The examples below cover FP8 and BF16 checkpoints.”

agentic scaffolding 1

Iterative tool execution with verification and revision core

“repeated tool calls, execution, verification, and revision”

2 more mentioned or evaluated
Attention data parallelism mentioned

serving parallelism “For more optimized configurations, including MegaMoE, attention data parallelism, and HiCache, see the [SGLang Cookbook for Step 5 Preview]”

HiCache mentioned

KV cache management “For more optimized configurations, including MegaMoE, attention data parallelism, and HiCache, see the [SGLang Cookbook for Step 5 Preview]”

filed at the root 1

Checkpoint sharding for offline generation used

“The standalone implementation converts checkpoints into shards for offline generation.”

kernel & quantization library 1

MegaMoE used

“This configuration uses MegaMoE for the FP8 routed experts.”

agent product 1

Repository-level software engineering with iterative code execution and verification core

“It can navigate large codebases, coordinate changes across files, run and verify code, and iterate on the results.”

infrastructure service 1

Automatic provider failover and retry used

“we can recover by routing to another healthy provider”

unfiled 3

Long-context understanding and reasoning core

“Analyze long documents, multiple source materials, and conversation history within a 1M-token context.”

Multi-step tool use for agent tasks core

“Use tools provided by an application to retrieve information, process documents, and continue through multi-step work.”

Tool use for multi-step tasks core

“Step 5 Preview can carry out tasks that require repeated tool calls, execution, verification, and revision”

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

SourceKindPublisherStatus
stepfun.com/step-5-previewofficial blogStepFunnot fetched
github.com/stepfun-ai/Step-5-Previewcode repoStepFunread
huggingface.co/stepfun-ai/Step-5-Previewmodel cardStepFunread
platform.stepfun.ai/docs/en/guides/models/step-5-previewvendor docsStepFunread
openrouter.ai/stepfun/step-5-previewvendor docsOpenRouterread
↑