Open Models Almanac
ModelsStep-3.7-Flash

StepFun

Step-3.7-Flash

What Step-3.7-Flash’s own documents say it is built from — every method with the sentence that describes it.

Parameters
~196B / ~11B active
Context
256K
Attention
sliding/global hybrid1 full : 3 sliding-window (512) GQA layers, gated attention
Experts
288 routed + 1 shared, top-8
Input
text, image
Licence
Apache 2.0
Released
2026-05-29

Against the consensus recipe

Its documents state 5 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 11 of the 19 architecture features checked — 9 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_step3p7.py, configuration_step3p7.py) and config.json at revision 5f6244077a, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention code only in its codenot stated

64 query heads share 8 key/value heads (96 query heads for sliding-attention layers also share 8 KV heads) via repeat_kv grouping.

show evidencetext_config.num_attention_heads = 64text_config.num_attention_groups = 8
self.num_key_value_groups = self.num_heads // config.num_key_value_heads
modeling_step3p7.py · L678
Multi-head latent attention not in its codenot stated

Attention uses separate q/k/v linear projections sized by head_dim with no kv_lora_rank/low-rank latent down-projection anywhere in the…

show the whole verdict

Attention uses separate q/k/v linear projections sized by head_dim with no kv_lora_rank/low-rank latent down-projection anywhere in the code or config.

Sliding-window attention code only in its codenot stated

layer_types marks most layers as sliding_attention with sliding_window=512, and Step3p7Attention sets self.sliding_window from…

show evidence

layer_types marks most layers as sliding_attention with sliding_window=512, and Step3p7Attention sets self.sliding_window from config.sliding_window when the layer is local.

text_config.sliding_window = 512text_config.layer_types = ["full_attention", "sliding_attention", "sliding_attention", "sliding_attenti…
self.is_local_attention = config.layer_types[layer_idx] == "sliding_attention"
        self.sliding_window = config.sliding_window if self.is_local_attention else None
modeling_step3p7.py · L693–694
Interleaved sliding-window and global attention code only in its codenot stated

layer_types interleaves one full_attention layer every 4 layers with three sliding_attention layers, and the model builds separate…

show evidence

layer_types interleaves one full_attention layer every 4 layers with three sliding_attention layers, and the model builds separate causal_mask_mapping entries for each.

text_config.layer_types = ["full_attention", "sliding_attention", "sliding_attention", "sliding_attenti…
causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "sliding_attention": create_sliding_window_causal_mask(**sliding_mask_kwargs),
            }
modeling_step3p7.py · L889–892
Indexer-selected sparse attention (DeepSeek Sparse Attention) code unclearnot stated

The config normalizes legacy sparse_attention_config keys into index_n_heads/index_head_dim/index_topk_blocks attributes and references a…

show evidence

The config normalizes legacy sparse_attention_config keys into index_n_heads/index_head_dim/index_topk_blocks attributes and references a 'sparse Lightning Indexer', but this config has no sparse_attention_config block and no indexer module is implemented in the given modeling file, so it cannot be confirmed active.

# `layer_types` is the canonical per-layer attention dispatch: it tells
        # `DynamicCache(config=...)` which layers want the sparse cache and tells
        # `Step3p7Attention` which layers build a sparse Lightning Indexer.
configuration_step3p7.py · L277–279
Linear-attention or state-space layers alongside full attention not in its codenot stated

All layer_types entries are either full_attention or sliding_attention softmax attention; no linear-attention or state-space layer type…

show evidence

All layer_types entries are either full_attention or sliding_attention softmax attention; no linear-attention or state-space layer type appears in this config's layer_types list.

text_config.layer_types = ["full_attention", "sliding_attention", "sliding_attention", "sliding_attenti…
Gated DeltaNet layers not in its codenot stated

No gated delta rule / recurrent state module exists in the provided code; layer_types contains only full_attention and sliding_attention.

Mamba-2 layers not in its codenot stated

No Mamba-2 SSD block or mamba-related config keys (mamba_num_heads, ssm_state_size) are present in the code or config.

Learnable attention sink not in its codenot stated

The config declares a 'sink' flag set to false and no per-head sink logit parameter or extra softmax column exists in the attention code.

show evidencetext_config.sink = false
Gated attention output code only in its codenot stated

The attention output is multiplied element-wise by a sigmoid of a learned gate projection (g_proj) before the output projection.

show evidencetext_config.use_head_wise_attn_gate = true
attn_output = attn_output.view(*output_shape[:-1], self.num_heads, self.head_dim)
        attn_output = attn_output * gate_states.unsqueeze(-1).sigmoid()
        attn_output = attn_output.view(*output_shape)
modeling_step3p7.py · L740–742
QK normalization code only in its codenot stated

Queries and keys are each passed through an RMSNorm (q_norm/k_norm) over the head dimension before rotary embedding and the attention dot…

show evidence

Queries and keys are each passed through an RMSNorm (q_norm/k_norm) over the head dimension before rotary embedding and the attention dot product.

query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
modeling_step3p7.py · L713–714
Partial RoPE code only in its codenot stated

partial_rotary_factors alternates 0.5 for full_attention layers and 1.0 for sliding_attention layers, and the rope init function scales…

show evidence

partial_rotary_factors alternates 0.5 for full_attention layers and 1.0 for sliding_attention layers, and the rope init function scales head_dim by this factor to compute the rotary dim.

text_config.partial_rotary_factors = [0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1…
partial_rotary_factor = config.rope_parameters[layer_type].get("partial_rotary_factor", 1.0)
        head_dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads
        dim = int(head_dim * partial_rotary_factor)
modeling_step3p7.py · L476–478
YaRN RoPE scaling not in its codenot stated

rope_scaling specifies rope_type 'llama3' (not YaRN) and the config has no YaRN rope_type entry; the 'yarn_only_types' key only names…

show evidence

rope_scaling specifies rope_type 'llama3' (not YaRN) and the config has no YaRN rope_type entry; the 'yarn_only_types' key only names which layer types the llama3-style scaling applies to.

text_config.rope_scaling = {"rope_type": "llama3", "factor": 2.0, "original_max_position_embeddings": 13…
Layers without positional encoding (NoPE) not in its codenot stated

Every layer_type (full_attention and sliding_attention) gets an entry in rope_parameters with rope_type 'default' plus rope_theta, so all…

show evidence

Every layer_type (full_attention and sliding_attention) gets an entry in rope_parameters with rope_type 'default' plus rope_theta, so all attention layers receive positional encoding; there is no no_rope_layers mechanism in the code.

text_config.use_rope_layers = []
params = {"rope_type": "default", "rope_theta": rope_theta[i]}
                if partial_rotary_factors:
                    params["partial_rotary_factor"] = partial_rotary_factors[i]
                self.rope_parameters[layer_type] = params
configuration_step3p7.py · L305–308
Mixture of experts in its codeadopted

Step3p7SparseMoeBlock routes tokens via Step3p7TopKRouter choosing top-8 of 288 experts, enabled on layers 3-44 per moe_layers_enum.

show evidencetext_config.use_moe = truetext_config.moe_num_experts = 288text_config.moe_top_k = 8
_, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False)
        top_k_weights = routing_weights.gather(1, top_k_index)
modeling_step3p7.py · L596–597
Shared expert code only in its codenot stated

Each sparse MoE block includes a Step3p7MLP shared expert of intermediate size share_expert_dim=1280 whose output is added unconditionally…

show evidence

Each sparse MoE block includes a Step3p7MLP shared expert of intermediate size share_expert_dim=1280 whose output is added unconditionally to the routed output.

text_config.share_expert_dim = 1280
self.shared_experts = Step3p7MLP(config, layer_idx, is_shared_expert=True)
modeling_step3p7.py · L610
hidden_states = hidden_states * self.routed_scaling_factor
        hidden_states = hidden_states + shared_output
modeling_step3p7.py · L621–622
Auxiliary-loss-free load balancing (selection bias) code only in its codenot stated

The router adds a learned e_score_correction_bias to the sigmoid routing scores only for top-k selection, then uses the un-biased…

show evidence

The router adds a learned e_score_correction_bias to the sigmoid routing scores only for top-k selection, then uses the un-biased routing_weights for weighting experts.

text_config.use_moe_router_bias = true
scores_for_choice = routing_weights + self.e_score_correction_bias
        _, top_k_index = torch.topk(scores_for_choice, self.top_k, dim=-1, sorted=False)
        top_k_weights = routing_weights.gather(1, top_k_index)
modeling_step3p7.py · L595–597
Multi-token prediction layers in its codeadopted

num_nextn_predict_layers is 3, and the text model's __init__ builds unexpected-key ignore patterns for 3 trailing MTP layers appended to…

show evidence

num_nextn_predict_layers is 3, and the text model's __init__ builds unexpected-key ignore patterns for 3 trailing MTP layers appended to the checkpoint.

text_config.num_nextn_predict_layers = 3
SwiGLU feed-forward code only in its codenot stated

Both the dense MLP and the MoE experts compute SiLU(gate_proj(x)) * up_proj(x) (hidden_act is normalized to 'silu'), matching SwiGLU.

show evidence
gate = self.act_fn(self.gate_proj(x)).clamp(max=self.limit)
        up = self.up_proj(x).clamp(min=-self.limit, max=self.limit)
        return self.down_proj(gate * up)
modeling_step3p7.py · L537–539

channel mixer 1

mixture of experts 1

Mixture of experts core

“Step 3.7 Flash is a 198B-parameter sparse Mixture-of-Experts (MoE) vision-language model”

prediction head 2

Multi-Token Prediction optional

“The latest NVFP4 checkpoint includes MTP draft layers and supports vLLM speculative decoding”

MTP speculative decoding optional

“supports vLLM speculative decoding with: --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'”

multimodal architecture 3

Language backbone core

“combines a 196B-parameter language backbone with a 1.8B-parameter vision encoder”

Vision encoder core

“combines a 196B-parameter language backbone with a 1.8B-parameter vision encoder for native image understanding”

Vision Transformer (ViT) used

“paired with a 1.88-billion-parameter ViT vision encoder”

context capacity 1

256K context window core

“Step 3.7 Flash supports a 256k context window”

Not used 1

Cascade attention not used

softmax attention “--disable-cascade-attn”

training precision 1

NVFP4 optional

“Step 3.7 Flash is also available in an NVFP4-quantized variant for efficient deployment on NVIDIA GPUs”

training parallelism 2

Expert parallelism used

“vllm serve ... --enable-expert-parallel”

Tensor parallelism used

“--tensor-parallel-size 8”

decoding strategy 2

EAGLE optional

“--speculative-algorithm EAGLE”

Multi-layer EAGLE optional

“--enable-multi-layer-eagle”

reasoning control 3

Reasoning parser used

“--reasoning-parser step3p5”

Step3p5 reasoning parser used

“--reasoning-parser step3p5”

Thinking mode optional

“offers three selectable reasoning levels (low, medium, and high) so developers can easily balance speed, cost, and cognitive depth”

inference quantization 11

FP8 used

“Released under Apache 2.0 with BF16, FP8, NVFP4, and GGUF weights on Hugging Face”

FP8 KV-cache quantization used

“requires modelopt activation and FP8 KV Cache alignment”

NVFP4 quantization used

“Released under Apache 2.0 with BF16, FP8, NVFP4, and GGUF weights on Hugging Face”

GGUF used

“Released under Apache 2.0 with BF16, FP8, NVFP4, and GGUF weights on Hugging Face”

ModelOpt NVFP4 re-quantization used

“For NVFP4 model ... --quantization modelopt”

FP16 multimodal projector used

“Multimodal Projector | FP16 | 3.97 GB”

ModelOpt FP4 quantization used

“--quantization modelopt_fp4”

NVIDIA ModelOpt quantization used

“--quantization modelopt”

IQ4_XS optional

“Language Model | IQ4_XS | 104.99 GB”

Q3_K_L optional

“Language Model | Q3_K_L | 102.5 GB”

Q4_K_S optional

“Language Model | Q4_K_S | 111.5 GB”

inference scheduling 1

Asynchronous inference and execution used

“--async-scheduling”

inference kernel 2

TensorRT-LLM multi-head attention backend used

“--attention-backend trtllm_mha”

FlashAttention-4 optional

“--mm-attention-backend fa4”

agentic scaffolding 6

Automatic tool choice used

“--enable-auto-tool-choice”

Python tool for visual tasks used

“we grant the model an enriched action space to interact with images, including cropping, zooming in and out, and drawing pixels or…”

Tool-call parser used

“--tool-call-parser step3p5”

Visual Search Tool used

“we strengthen its ability to invoke the Visual Search tool”

step3p5 tool-call parser used

“--tool-call-parser step3p5”

Advisor strategy optional

“To push quality further without giving up Flash-tier efficiency, Step 3.7 Flash supports Advisor Mode.”

inference engine 1

llama.cpp used

“use llama.cpp”

kernel & quantization library 2

FlashInfer TensorRT-LLM backend used

“--moe-runner-backend flashinfer_trtllm”

FlashInfer backend optional

“--mm-attention-backend fa4”

unfiled 4

Apache 2.0 License used

“Released under Apache 2.0 with BF16, FP8, NVFP4, and GGUF weights on Hugging Face”

BF16 (bfloat16) used

“Released under Apache 2.0 with BF16, FP8, NVFP4, and GGUF weights on Hugging Face”

GUI operation used

“We extend Step 3.7 Flash with GUI operation, in particular for the Phone-use stack”

1 more mentioned or evaluated
Emergent Compositional Tool Use mentioned

“the model combined visual tools with non-visual tools without being explicitly trained to do so”

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

↑