StepFun
Step-3.7-Flash
What Step-3.7-Flash’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- ~196B / ~11B active
- Context
- 256K
- Attention
- sliding/global hybrid1 full : 3 sliding-window (512) GQA layers, gated attention
- Experts
- 288 routed + 1 shared, top-8
- Input
- text, image
- Licence
- Apache 2.0
- Released
- 2026-05-29
Against the consensus recipe
Its documents state 5 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictionoptional
- Grouped-query attentionnot in its documents
Optimization
- Muonnot in its documents
- Expert parallelismused
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modeoptional
- Multi-Token Predictionnot in its documents
- FP8used
Checked in its code
Its code shows 11 of the 19 architecture features checked — 9 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_step3p7.py, configuration_step3p7.py) and config.json at revision 5f6244077a, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention code only | in its code | not stated | 64 query heads share 8 key/value heads (96 query heads for sliding-attention layers also share 8 KV heads) via repeat_kv grouping. show evidencetext_config.num_attention_heads = 64text_config.num_attention_groups = 8 |
| Multi-head latent attention | not in its code | not stated | Attention uses separate q/k/v linear projections sized by head_dim with no kv_lora_rank/low-rank latent down-projection anywhere in the… show the whole verdictAttention uses separate q/k/v linear projections sized by head_dim with no kv_lora_rank/low-rank latent down-projection anywhere in the code or config. |
| Sliding-window attention code only | in its code | not stated | layer_types marks most layers as sliding_attention with sliding_window=512, and Step3p7Attention sets self.sliding_window from… show evidencelayer_types marks most layers as sliding_attention with sliding_window=512, and Step3p7Attention sets self.sliding_window from config.sliding_window when the layer is local. text_config.sliding_window = 512text_config.layer_types = ["full_attention", "sliding_attention", "sliding_attention", "sliding_attenti… |
| Interleaved sliding-window and global attention code only | in its code | not stated | layer_types interleaves one full_attention layer every 4 layers with three sliding_attention layers, and the model builds separate… show evidencelayer_types interleaves one full_attention layer every 4 layers with three sliding_attention layers, and the model builds separate causal_mask_mapping entries for each. text_config.layer_types = ["full_attention", "sliding_attention", "sliding_attention", "sliding_attenti… |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | code unclear | not stated | The config normalizes legacy sparse_attention_config keys into index_n_heads/index_head_dim/index_topk_blocks attributes and references a… show evidenceThe config normalizes legacy sparse_attention_config keys into index_n_heads/index_head_dim/index_topk_blocks attributes and references a 'sparse Lightning Indexer', but this config has no sparse_attention_config block and no indexer module is implemented in the given modeling file, so it cannot be confirmed active. |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | All layer_types entries are either full_attention or sliding_attention softmax attention; no linear-attention or state-space layer type… show evidenceAll layer_types entries are either full_attention or sliding_attention softmax attention; no linear-attention or state-space layer type appears in this config's layer_types list. text_config.layer_types = ["full_attention", "sliding_attention", "sliding_attention", "sliding_attenti… |
| Gated DeltaNet layers | not in its code | not stated | No gated delta rule / recurrent state module exists in the provided code; layer_types contains only full_attention and sliding_attention. |
| Mamba-2 layers | not in its code | not stated | No Mamba-2 SSD block or mamba-related config keys (mamba_num_heads, ssm_state_size) are present in the code or config. |
| Learnable attention sink | not in its code | not stated | The config declares a 'sink' flag set to false and no per-head sink logit parameter or extra softmax column exists in the attention code. show evidencetext_config.sink = false |
| Gated attention output code only | in its code | not stated | The attention output is multiplied element-wise by a sigmoid of a learned gate projection (g_proj) before the output projection. show evidencetext_config.use_head_wise_attn_gate = true |
| QK normalization code only | in its code | not stated | Queries and keys are each passed through an RMSNorm (q_norm/k_norm) over the head dimension before rotary embedding and the attention dot… show evidenceQueries and keys are each passed through an RMSNorm (q_norm/k_norm) over the head dimension before rotary embedding and the attention dot product. |
| Partial RoPE code only | in its code | not stated | partial_rotary_factors alternates 0.5 for full_attention layers and 1.0 for sliding_attention layers, and the rope init function scales… show evidencepartial_rotary_factors alternates 0.5 for full_attention layers and 1.0 for sliding_attention layers, and the rope init function scales head_dim by this factor to compute the rotary dim. text_config.partial_rotary_factors = [0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1… |
| YaRN RoPE scaling | not in its code | not stated | rope_scaling specifies rope_type 'llama3' (not YaRN) and the config has no YaRN rope_type entry; the 'yarn_only_types' key only names… show evidencerope_scaling specifies rope_type 'llama3' (not YaRN) and the config has no YaRN rope_type entry; the 'yarn_only_types' key only names which layer types the llama3-style scaling applies to. text_config.rope_scaling = {"rope_type": "llama3", "factor": 2.0, "original_max_position_embeddings": 13… |
| Layers without positional encoding (NoPE) | not in its code | not stated | Every layer_type (full_attention and sliding_attention) gets an entry in rope_parameters with rope_type 'default' plus rope_theta, so all… show evidenceEvery layer_type (full_attention and sliding_attention) gets an entry in rope_parameters with rope_type 'default' plus rope_theta, so all attention layers receive positional encoding; there is no no_rope_layers mechanism in the code. text_config.use_rope_layers = [] |
| Mixture of experts | in its code | adopted | Step3p7SparseMoeBlock routes tokens via Step3p7TopKRouter choosing top-8 of 288 experts, enabled on layers 3-44 per moe_layers_enum. show evidencetext_config.use_moe = truetext_config.moe_num_experts = 288text_config.moe_top_k = 8 |
| Shared expert code only | in its code | not stated | Each sparse MoE block includes a Step3p7MLP shared expert of intermediate size share_expert_dim=1280 whose output is added unconditionally… show evidenceEach sparse MoE block includes a Step3p7MLP shared expert of intermediate size share_expert_dim=1280 whose output is added unconditionally to the routed output. text_config.share_expert_dim = 1280 |
| Auxiliary-loss-free load balancing (selection bias) code only | in its code | not stated | The router adds a learned e_score_correction_bias to the sigmoid routing scores only for top-k selection, then uses the un-biased… show evidenceThe router adds a learned e_score_correction_bias to the sigmoid routing scores only for top-k selection, then uses the un-biased routing_weights for weighting experts. text_config.use_moe_router_bias = true |
| Multi-token prediction layers | in its code | adopted | num_nextn_predict_layers is 3, and the text model's __init__ builds unexpected-key ignore patterns for 3 trailing MTP layers appended to… show evidencenum_nextn_predict_layers is 3, and the text model's __init__ builds unexpected-key ignore patterns for 3 trailing MTP layers appended to the checkpoint. text_config.num_nextn_predict_layers = 3 |
| SwiGLU feed-forward code only | in its code | not stated | Both the dense MLP and the MoE experts compute SiLU(gate_proj(x)) * up_proj(x) (hidden_act is normalized to 'silu'), matching SwiGLU. show evidence |
channel mixer 1
mixture of experts 1
“Step 3.7 Flash is a 198B-parameter sparse Mixture-of-Experts (MoE) vision-language model”
prediction head 2
“The latest NVFP4 checkpoint includes MTP draft layers and supports vLLM speculative decoding”
“supports vLLM speculative decoding with: --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'”
multimodal architecture 3
“combines a 196B-parameter language backbone with a 1.8B-parameter vision encoder”
“combines a 196B-parameter language backbone with a 1.8B-parameter vision encoder for native image understanding”
“paired with a 1.88-billion-parameter ViT vision encoder”
context capacity 1
“Step 3.7 Flash supports a 256k context window”
Not used 1
softmax attention “--disable-cascade-attn”
training precision 1
“Step 3.7 Flash is also available in an NVFP4-quantized variant for efficient deployment on NVIDIA GPUs”
training parallelism 2
“vllm serve ... --enable-expert-parallel”
“--tensor-parallel-size 8”
decoding strategy 2
“--speculative-algorithm EAGLE”
“--enable-multi-layer-eagle”
reasoning control 3
“--reasoning-parser step3p5”
“--reasoning-parser step3p5”
“offers three selectable reasoning levels (low, medium, and high) so developers can easily balance speed, cost, and cognitive depth”
inference quantization 11
“Released under Apache 2.0 with BF16, FP8, NVFP4, and GGUF weights on Hugging Face”
“requires modelopt activation and FP8 KV Cache alignment”
“Released under Apache 2.0 with BF16, FP8, NVFP4, and GGUF weights on Hugging Face”
“Released under Apache 2.0 with BF16, FP8, NVFP4, and GGUF weights on Hugging Face”
“For NVFP4 model ... --quantization modelopt”
“Multimodal Projector | FP16 | 3.97 GB”
“--quantization modelopt_fp4”
“--quantization modelopt”
“Language Model | IQ4_XS | 104.99 GB”
“Language Model | Q3_K_L | 102.5 GB”
“Language Model | Q4_K_S | 111.5 GB”
inference scheduling 1
“--async-scheduling”
inference kernel 2
“--attention-backend trtllm_mha”
“--mm-attention-backend fa4”
agentic scaffolding 6
“--enable-auto-tool-choice”
“we grant the model an enriched action space to interact with images, including cropping, zooming in and out, and drawing pixels or…”
“--tool-call-parser step3p5”
“we strengthen its ability to invoke the Visual Search tool”
“--tool-call-parser step3p5”
“To push quality further without giving up Flash-tier efficiency, Step 3.7 Flash supports Advisor Mode.”
inference engine 1
“use llama.cpp”
kernel & quantization library 2
“--moe-runner-backend flashinfer_trtllm”
“--mm-attention-backend fa4”
unfiled 4
“Released under Apache 2.0 with BF16, FP8, NVFP4, and GGUF weights on Hugging Face”
“Released under Apache 2.0 with BF16, FP8, NVFP4, and GGUF weights on Hugging Face”
“We extend Step 3.7 Flash with GUI operation, in particular for the Phone-use stack”
1 more mentioned or evaluated
“the model combined visual tools with non-visual tools without being explicitly trained to do so”
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| static.stepfun.com/blog/step-3.7-flash | official blog | StepFun | read |
| github.com/stepfun-ai/Step-3.7-Flash | code repo | StepFun | read |
| huggingface.co/stepfun-ai/Step-3.7-Flash | model card | StepFun | read |
| platform.stepfun.ai/docs/en/guides/models/step-3.7-flash | vendor docs | StepFun | read |
| marktechpost.com/2026/05/29/stepfun-releases-step-3-7-flash-a-198b-moe-… | news | MarkTechPost | read |
| finance.biggo.com/news/B2LfcZ4BoQmpnl36aGPg | news | BigGo Finance | read |