Alibaba (Qwen)
Qwen3.8-Flash-Next
What Qwen3.8-Flash-Next’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 125B / 6B active
- Context
- 256K
- Attention
- linear hybrid3:1 Gated DeltaNet : Qwen Sparse Attention (block-level, lightweight indexer)
- Experts
- 512 routed + 1 shared, top-10; plus 51B n-gram embedding params
- Input
- text, image, video
- Licence
- Qwen Community License 1.0
- Released
- 2026-08-26
Against the consensus recipe
Its documents state 5 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionused
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictioncore
- Grouped-query attentionnot in its documents
Optimization
- Muoncore
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modenot in its documents
- Multi-Token Predictionoptional
- FP8not in its documents
Checked in its code
Its code shows 11 of the 19 architecture features checked — 7 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_qwen4_exp.py, configuration_qwen4_exp.py) and config.json at revision 812d96e10a, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention code only | in its code | not stated | Full-attention (indexed_attention) layers project 24 query heads but only 2 key/value heads, repeated via repeat_kv with… show evidenceFull-attention (indexed_attention) layers project 24 query heads but only 2 key/value heads, repeated via repeat_kv with num_key_value_groups. text_config.num_attention_heads = 24text_config.num_key_value_heads = 2 |
| Multi-head latent attention | not in its code | not stated | No kv_lora_rank/q_lora_rank or latent KV down/up-projection is present; attention uses standard per-head k_proj/v_proj instead. |
| Sliding-window attention | not in its code | not stated | There is no sliding_window config key or windowed mask logic in the attention/masking code for this model; layer_types only distinguishes… show the whole verdictThere is no sliding_window config key or windowed mask logic in the attention/masking code for this model; layer_types only distinguishes linear_attention and indexed_attention (full attention with an indexer), with no local window. |
| Interleaved sliding-window and global attention | not in its code | not stated | layer_types interleaves linear_attention with full/indexed attention, not sliding-window with global attention, so there is no… show evidencelayer_types interleaves linear_attention with full/indexed attention, not sliding-window with global attention, so there is no sliding-window/global interleaving. text_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"… |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) code only | in its code | not stated | A Qwen4ExpTextQSAIndexer scores compressed key blocks per query head and selects a top-k token budget, masking attention to only those… show evidenceA Qwen4ExpTextQSAIndexer scores compressed key blocks per query head and selects a top-k token budget, masking attention to only those tokens, enabled via indexer_n_heads/indexer_budget/indexer_compress_ratio in the config. text_config.indexer_n_heads = 4text_config.indexer_budget = 2048text_config.indexer_compress_ratio = 4 |
| Linear-attention or state-space layers alongside full attention code only | in its code | not stated | layer_types mixes linear_attention (Gated DeltaNet) layers with full/indexed-attention softmax layers in a fixed pattern. show evidencetext_config.layer_types = ["linear_attention", "linear_attention", "linear_attention", "full_attention"… |
| Gated DeltaNet layers | in its code | adopted | Qwen4ExpTextGatedDeltaNet implements the gated delta rule with a decay gate (A_log/dt_bias) and delta-rule state update, used on… show evidenceQwen4ExpTextGatedDeltaNet implements the gated delta rule with a decay gate (A_log/dt_bias) and delta-rule state update, used on linear_attention layers. text_config.linear_num_key_heads = 16text_config.linear_conv_kernel_dim = 4 |
| Mamba-2 layers | not in its code | not stated | No Mamba-2/SSD selective state-space block is implemented; the recurrent mixer present is the gated delta rule (GatedDeltaNet), not Mamba-2. |
| Learnable attention sink | not in its code | not stated | No per-head learned sink logit is added to the attention softmax anywhere in the code. |
| Gated attention output code only | in its code | not stated | Attention output is multiplied by a sigmoid gate derived from a chunk of the q_proj output before the output projection. show evidence |
| QK normalization code only | in its code | not stated | Queries and keys are each passed through an RMSNorm (q_norm/k_norm) per head before RoPE and the attention dot product. show evidence |
| Partial RoPE | in its code | adopted | partial_rotary_factor of 0.25 restricts RoPE to a quarter of the head dimension, with the rest (q_nope/k_nope) concatenated back unrotated. show evidencetext_config.partial_rotary_factor = 0.25text_config.rope_parameters.partial_rotary_factor = 0.25 |
| YaRN RoPE scaling documents disagree | not in its code | adopted | rope_parameters.rope_type is set to "default", not YaRN, so no YaRN scaling is applied. show evidencetext_config.rope_parameters.rope_type = "default" |
| Layers without positional encoding (NoPE) | not in its code | not stated | All softmax-attention (indexed_attention/full_attention) layers apply the same partial RoPE via apply_rotary_pos_emb; there is no… show the whole verdictAll softmax-attention (indexed_attention/full_attention) layers apply the same partial RoPE via apply_rotary_pos_emb; there is no per-layer toggle disabling RoPE entirely on some attention layers (linear_attention layers are not softmax attention and don't count). |
| Mixture of experts | in its code | adopted | The MLP block routes each token to num_experts_per_tok of num_experts experts via a learned TopKRouter. show evidencetext_config.num_experts = 512text_config.num_experts_per_tok = 10 |
| Shared expert code only | in its code | not stated | Each MoE block has a shared_expert MLP with nonzero shared_expert_intermediate_size applied to every token and gated before being added to… show evidenceEach MoE block has a shared_expert MLP with nonzero shared_expert_intermediate_size applied to every token and gated before being added to routed expert output. text_config.shared_expert_intermediate_size = 640 |
| Auxiliary-loss-free load balancing (selection bias) | not in its code | not stated | The router in Qwen4ExpTextTopKRouter computes plain softmax + topk with no per-expert correction bias (e_score_correction_bias) added to… show evidenceThe router in Qwen4ExpTextTopKRouter computes plain softmax + topk with no per-expert correction bias (e_score_correction_bias) added to the scores. |
| Multi-token prediction layers | in its code | adopted | The config declares an mtp block with num_hidden_layers=1 (also mtp_num_hidden_layers=1), i.e. extra next-token prediction layer(s) are… show evidenceThe config declares an mtp block with num_hidden_layers=1 (also mtp_num_hidden_layers=1), i.e. extra next-token prediction layer(s) are declared in config. text_config.mtp_num_hidden_layers = 1text_config.mtp.num_hidden_layers = 1 |
| SwiGLU feed-forward code only | in its code | not stated | Both the dense MLP and expert MLPs compute silu(gate_proj(x)) * up_proj(x), i.e. SwiGLU, matching hidden_act=silu. show evidencetext_config.hidden_act = "silu" |
token mixer 14
sparse attention 11
“it aggregates the sequence into micro-blocks, estimates importance at the block level, and then selects the most relevant regions”
“Attention cost stops growing with context length once the budget binds”
“uses compressed QSA with a compression ratio of 4, or c4.”
“During the final stage of CPT, QSA is enabled and the backbone and indexer are jointly trained for 8,000 steps”
block selection 2
“QSA keeps the best 512 blocks, expands them back to 2048 logical token positions”
“each query selects the highest-scoring compressed blocks”
sparse attention indexer 5
“So the draft decode steps stop running the indexer altogether.”
“uses a compressed lightweight indexer to select the important context at micro-block granularity”
“The selection machinery is small and explicit: an MQA indexer with 4 query heads and a single shared key head, head dimension 128…”
“keys are partitioned into non-overlapping blocks of tokens and compressed by average pooling”
“Block-level importance scores are then obtained through block-causal scoring.”
linear attention & state space 2
gated delta network 2
“GDN compresses the prefix into a fixed-size recurrent state and updates that state according to the current content”
“Unlike the original GDN, which uses a SiLU output gate, we use the bounded sigmoid gate”
hybrid layer stacking 1
“The 3:1 interleave of Gated DeltaNet to full-attention layers is the same ratio Qwen3.5 uses”
channel mixer 3
“The embeddings are further injected into residual stream through contextual gating mechanism”
mixture of experts 2
“Qwen3.8-Flash-Next is a sparse mixture-of-experts model”
positional encoding 3
“All the notable open-source frameworks implement static YaRN, which means the scaling factor remains constant regardless of input length”
“partial RoPE is applied to 64 of the 128 dimensions in each indexer head”
“For long-horizon tasks where the total length (including both input and output) exceeds this limit, we recommend using RoPE scaling…”
normalization & residual 14
“the gate decides how that capacity is spent, while also supplying the rescaling that keeps training stable”
“The block output is written to every branch through one data-dependent scalar per branch”
“It then predicts elementwise gating scores per branch and channel from all branches”
“controls reads and writes with a dynamic gate”
“The read gate is element-wise and data-dependent”
“widens the residual stream into 4 branches”
“Combine computes four injection coefficients and applies a residual update to the four streams.”
“the multiplicative gate supplies the necessary rescaling directly, keeping the training stable”
“GR first normalizes each branch independently,”
“Mix uses a low-rank projection to generate element-wise gates and reduce the four residual streams into one hidden state.”
“The residual stream is widened to four branches”
“A sigmoid gate is better than in both loss and training stability.”
“Making and data-dependent reduces loss by 0.002 over the static variant”
“we adopt zero-centered RMSNorm to constrain the growth of RMSNorm weights”
prediction head 1
“MTP: 1 layer, trained with multi-steps”
context capacity 7
“short n-grams ending at each token serve as keys into embedding tables, and the retrieved vectors augment the corresponding token…”
“Capacity is further added outside the backbone by a single n-gram embedding layer whose tables are prefetched from host memory”
“Twenty million bigram and trigram embeddings are applied at layer 2, contributing 51B parameters — more than 40% of the checkpoint's…”
“A single N-gram embedding layer is sufficient. ... We place it at Layer 2”
“The N-gram Embedding can reside entirely in host memory to save GPU memory”
“lookup positions are computed in advance and asynchronously prefetched, so it never permanently occupies GPU memory.”
“We use multi-head hashing for embedding lookup”
5 more mentioned or evaluated
normalization & residual “mHC (Xie et al., 2025) uses a sigmoid and additionally constrains to a manifold of doubly stochastic matrices.”
normalization & residual “HC generalizes Eq. (21) and Eq. (22) into three learnable operators.”
mixture of experts “frequency-based partitioning of embedding slots. Despite these efforts, we observed no consistent performance gains in our training recipe.”
normalization & residual “We first study how much of the reported gain comes from width alone, using a simplified variant of AltUp”
context capacity “including but not limited to token normalization for vocabulary compression ... Despite these efforts, we observed no consistent…”
Not used 3
dense feed-forward network “This allows stable training without explicit activation control such as qk-clip and SwiGLU-clip”
normalization & residual “This allows stable training without explicit activation control such as qk-clip”
normalization & residual “each block reads only the two branches with the highest gate values instead of all. Pre-training loss and benchmarks were almost…”
multi-token prediction objective 1
“MTP: 1 layer, trained with multi-steps”
distillation objective 1
“Stage 1: Dense Distillation. We first distill the full-sequence attention distribution of the backbone into the indexer.”
optimization 27
filed at the root 2
“Therefore, we develop an updated hyperparameter scaling law.”
“Phase-by-phase cost accounting directed the sparse-attention design into the indexer”
optimizer 15
“We use Muon as the main optimizer”
“Muon exacerbates early-training fluctuations and destabilizes the router. ... Therefore, we use AdamW for the router.”
“The Muon and AdamW optimizers are applied to specific weight categories to maximize efficiency.”
“We therefore split the fused gradient before orthogonalization, run NS on each sub-matrix independently, and gather the results back into…”
“We apply Muon to the two-dimensional weights that genuinely act as linear maps”
“We set the Newton–Schulz iteration to 8 steps, favouring the additional stability under stress.”
“the -gram embedding table runs on Adam with weight decay disabled.”
“The NS iteration is applied to a Nesterov-accelerated momentum”
“an asynchronous Micro-Group pipeline reconstructs each Muon-owned matrix via fused All-to-All across TP ranks.”
“We developed Canzona, which decouples logical optimizer assignment from physical parameter layout”
“We capture the whole step in a CUDA graph to remove this overhead.”
“refined around orthogonalization accuracy”
“The and GDN input projections are split at per-head granularity, which improves both loss and downstream benchmarks.”
“we adopt the per-step coefficient schedule of the Polar Express method”
“The fc1 is split into its gate and up halves; the loss is essentially unchanged while benchmarks improve slightly.”
learning-rate schedule 3
“we eliminate traditional batch-size warmups and start directly at the target batch size”
“we hold the learning rate constant at a multiple of its optimal value, bypassing the standard decay schedule to simulate the prolonged…”
“Downstream benchmark results demonstrate the robust convergence achieved by our new scaling-law fit”
training precision 1
“Storing the branches in FP8 halves the bytes moved for the residual state relative to BF16, with almost no loss in quality.”
training stability 2
“large-scale instabilities can be reproduced in small models by raising the learning rate”
“we design a set of stress tests that amplify the relevant stress within a smaller budget”
training parallelism 2
“Use TEP8 with the Triton MoE backend on Hopper”
“an -balanced static partitioner reassigns whole parameters (no cut inside a tensor) to equalize estimated NS FLOPs across DP ranks”
1 more mentioned or evaluated
learning-rate schedule “the first maintains the peak learning rate of the constant-batch optimum, making the ramp its only difference”
Not used 1
training stability “without relying on explicit clipping methods such as qk-clip or SwiGLU-clip”
decoding strategy 2
“the model ships a 1-layer MTP head trained with multiple steps, exposed by SGLang as native NEXTN speculative decoding.”
“the built-in Multi-Token Prediction module supports speculative decoding.”
reasoning control 1
“Qwen3.8-Flash-Next models operate in thinking mode by default”
KV cache management 1
“We place it at Layer 2, allowing host-memory prefetching to overlap with the computation of the first layer.”
inference quantization 1
“an FP8 checkpoint that uses fine-grained quantization with a block size of 128”
serving parallelism 2
“vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \ --tensor-parallel-size 4”
“Enable expert parallelism with --enable-expert-parallel to improve throughput.”
inference scheduling 1
“transformers serve Qwen/Qwen3.8-Flash-Next --port 8000 --continuous-batching”
inference kernel 7
“Shape-aware dispatch lets HC use the appropriate execution path for both low-latency decode and large-scale prefill.”
“SGLang keeps each rank's vocabulary-parallel table shard in pinned host memory and gathers the selected rows into a small BF16 GPU buffer…”
“the step is captured in a CUDA graph”
“we implement a fused QSA kernel that jointly computes sparse attention outputs and the KL loss without materializing intermediate results”
“SiLU, Sigmoid, gating, and the final reduction are fused into the two GEMM epilogues, avoiding intermediate writes to global memory.”
“At small M, this mapping exposes too few CTAs, so the M ≤ 32 path splits each row along the hidden dimension.”
“For M ≤ 16, we use the low-latency split-K CuTe GEMM from FlashInfer PR #4266.”
context management 1
“By default, Qwen3.8-Flash-Next retains thinking blocks from all historical messages, maintaining a complete reasoning trace across the…”
kernel & quantization library 1
“we optimize the GDN kernel with FlashQLA, a TileLang-based fused linear-attention kernel library”
filed at the root 2
“runs therefore apply a fixed acceptance length measured separately”
“Where pre-training metrics agreed but late-stage evaluation diverged, the joint protocol caught the discrepancy before it reached production”
unfiled 23
“out of every 4 layers, 3 GDN layers compress history into a fixed-size state, while the remaining layer performs precise retrieval over…”
“Token mixing uses a layer-wise hybrid of Gated DeltaNet (GDN) and global attention”
“three of every four layers use Gated DeltaNet to compress history; the fourth uses Qwen Sparse Attention for precise long-range retrieval.”
“the QSA top-k selection computed by the draft-extend pass over the tokens the target just accepted is held for the whole MTP iteration”
“QSA employs a lightweight indexer that compresses the sequence into micro-block representations, estimates their importance, and selects…”
“The efficiency advantage of GR is that removing eliminates a full read of the residual state per block”
“We place one such full-attention layer in every four layers, with the other three layers using GDN.”
“the recipe enables PLE CPU offload automatically on this hardware.”
“we can recover by routing to another healthy provider”
“the read of Eq. (30)–(32) and the write of Eq. (33)–(34) are each fused into a single kernel, with the group RMSNorm folded into the read”
“Fusing the residual read and the final normalization before the LM head into one gated read operation further reduces the gradient norm”
“Mamba-cache capacity error at startup: keep --max-num-seqs 256.”
“we apply max pooling to align this distribution with the block-level indexer scores”
“and the per-block maximum activation”
“the of the pre-clip gradient norm and the number of threshold crossings”
“The up-projection weights are reordered offline so the four gate values for each output can be reduced locally inside a tile.”
“A dedicated CUDA stream overlaps the gather with the first decoder block.”
“if your request filters allow it”
“loss spikes (steps exceeding a -step rolling median by more than )”
“It is recommended to set the longest_edge parameter in the video_preprocessor_config file to 469,762,048 (corresponding to 224k video…”
2 more mentioned or evaluated
“including but not limited to token normalization for vocabulary compression ..., non-uniform allocation across N-gram orders, and…”
“Validation used TieringOffloadingSpec: 8 GiB total CPU tier, blocks_per_chunk: 1”
Not used 1
“each block reads only the two branches with the highest gate values instead of all”
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| qwen.ai/blog?id=qwen3.8-flash-next | official blog | Qwen | not fetched |
| arxiv.org/abs/2608.30320 | technical report | Qwen | read |
| huggingface.co/Qwen/Qwen3.8-Flash-Next | model card | Qwen | read |
| github.com/QwenLM/Qwen3.8-Flash-Next | code repo | Qwen | read |
| lmsys.org/blog/2026-08-26-qwen-flash-next | vendor docs | LMSYS | read |
| recipes.vllm.ai/Qwen/Qwen3.8-Flash-Next | vendor docs | vLLM | read |
| inferencex.semianalysis.com/model/qwen-3-8-flash-next | third party analysis | SemiAnalysis | read |
| openrouter.ai/qwen/qwen3.8-flash | vendor docs | OpenRouter | read |