Model techniques map
Modelsgpt-oss-120b

OpenAI · released 2025-08-05

gpt-oss-120b

What gpt-oss-120b’s own documents say it is built from — every method with the sentence that describes it.

OpenRouterHugging Face

Curator’s note117B total / 5.1B active MoE, Apache 2.0, MXFP4-quantized MoE weights so the 120b fits on one 80GB GPU, 128K context, harmony response format, configurable reasoning effort. The oldest model in the map and the only one from 2025: it is here because a year after release it is still in the weekly open-weight top 20 (rank 17), which makes it the map's baseline for what 2025 practice looked like. Its 'technical report' is the published model card (arXiv 2508.10925).

Against the consensus recipe

Its documents state 3 of the 15 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something the documents do not say, not something the model lacks.

  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 7 of the 19 architecture features checked — 3 of them are not stated in its documents.

Documents say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_gpt_oss.py, configuration_gpt_oss.py) and config.json at revision b5c939de8f, by anthropic/claude-sonnet-5 on 2026-09-25.

FeatureCodeIts documentsEvidence
Grouped-query attention in its code used

64 query heads share 8 key/value heads via repeat_kv grouping in every attention layer.

num_attention_heads = 64num_key_value_heads = 8
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_gpt_oss.py · L276
Multi-head latent attention not in its codenot stated

Attention uses standard separate q/k/v projections with no low-rank latent KV cache mechanism.

Sliding-window attention code only in its codenot stated

Layers marked sliding_attention use a 128-token sliding window mask.

sliding_window = 128layer_types = ["sliding_attention", "full_attention", "sliding_attention", "full_attention"…
self.sliding_window = config.sliding_window if self.layer_type == "sliding_attention" else None
modeling_gpt_oss.py · L292
Interleaved sliding-window and global attention code only in its codenot stated

layer_types alternates sliding_attention and full_attention across the 36 layers, with separate masks built for each.

layer_types = ["sliding_attention", "full_attention", "sliding_attention", "full_attention"…
causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "sliding_attention": create_sliding_window_causal_mask(**mask_kwargs),
            }
modeling_gpt_oss.py · L473–476
Indexer-selected sparse attention (DeepSeek Sparse Attention) not in its codenot stated

No indexer-based top-k token selection mechanism exists in this attention implementation.

Linear-attention or state-space layers alongside full attention not in its codenot stated

All token-mixing layers are softmax attention (full or sliding); there are no linear-attention or SSM layers.

Gated DeltaNet layers not in its codenot stated

No gated delta-rule recurrent layers are implemented; all layers use GptOssAttention.

Mamba-2 layers not in its codenot stated

No Mamba-2 SSM blocks are present; the model is a pure transformer with MoE FFNs.

Learnable attention sink code only in its codenot stated

Each attention layer has a learned per-head sink logit concatenated into the softmax denominator.

sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_gpt_oss.py · L251–252
self.sinks = nn.Parameter(torch.empty(config.num_attention_heads))
modeling_gpt_oss.py · L293
Gated attention output not in its codenot stated

The attention output is passed straight to o_proj with no sigmoid gating multiplication.

QK normalization not in its codenot stated

Query and key states go directly from projection to RoPE application with no normalization step.

Partial RoPE not in its codenot stated

Rotary embedding is applied to the full head dimension via chunking into two halves, not a partial slice.

first_half, second_half = torch.chunk(x, 2, dim=-1)
    first_ = first_half * cos - second_half * sin
    second_ = second_half * cos + first_half * sin
    return torch.cat((first_, second_), dim=-1)
modeling_gpt_oss.py · L220–223
YaRN RoPE scaling in its code used

rope_scaling specifies rope_type yarn with factor 32 and beta parameters, applied via ROPE_INIT_FUNCTIONS.

rope_scaling.rope_type = "yarn"rope_scaling.factor = 32.0
self.rope_type = self.config.rope_parameters["rope_type"]
        rope_init_fn: Callable = self.compute_default_rope_parameters
        if self.rope_type != "default":
            rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
modeling_gpt_oss.py · L158–161
Layers without positional encoding (NoPE) not in its codenot stated

Rotary position embeddings are applied uniformly to every attention layer's queries and keys; there is no layer-conditional skip of RoPE.

cos, sin = position_embeddings
        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
modeling_gpt_oss.py · L310–311
Mixture of experts in its code core

Each token is routed via top-k (4 of 128) learned router to expert MLPs.

num_local_experts = 128num_experts_per_tok = 4
router_top_value, router_indices = torch.topk(router_logits, self.top_k, dim=-1)  # (num_tokens, top_k)
modeling_gpt_oss.py · L128
Shared expert not in its codenot stated

There is no shared-expert module in GptOssMLP or its config; only routed experts exist.

Auxiliary-loss-free load balancing (selection bias) not in its codenot stated

The router uses a standard softmax top-k with a learned bias term added before topk, but there is no e_score_correction_bias style selection-only bias; instead an auxiliary load-balancing loss (router_aux_loss_coef) is used for training.

router_aux_loss_coef = 0.9
router_logits = F.linear(hidden_states, self.weight, self.bias)  # (num_tokens, num_experts)
modeling_gpt_oss.py · L127
Multi-token prediction layers not in its codenot stated

No multi-token-prediction modules or config fields are declared in this model.

SwiGLU feed-forward in its code used

Expert MLPs compute a SiLU-based gated GLU (sigmoid-approximated SiLU gate times up-projection) as the feed-forward activation.

hidden_act = "silu"
glu = gate * torch.sigmoid(gate * self.alpha)
        gated_output = (up + 1) * glu
modeling_gpt_oss.py · L86–87

model architecture 10

token mixer 3

softmax attention 2

Learned softmax denominator bias used

grouped-query attention 1

Grouped-query attention used

hybrid layer stacking 1

Hybrid Attention used

channel mixer 3

dense feed-forward network 1

SwiGLU used

positional encoding 2

YaRN used
Rotary Position Embedding used

normalization & residual 2

RMSNorm used
Pre-LN used

optimization 1

quantization-aware training 1

FP4 Quantization default

data curation 4

data filtering 1

CBRN pre-training data filtering used

data mixture & curriculum 2

Text-only pre-training corpus used
ProtocolQA-aligned RL training datasets evaluated

tokenization 1

o200k_harmony tokenizer used

post-training 9

supervised fine-tuning 3

Adversarial fine-tuning used
Instruction hierarchy training used
Fine-tuning optional

reinforcement learning algorithm 3

Chain-of-Thought Reinforcement Learning core
Helpful-Only Adversarial Reinforcement Learning used
Incremental Reinforcement Learning used

preference optimization 1

Deliberative alignment used

agentic post-training 2

Agentic tool-use training used
Python tool use in chain-of-thought used

inference & serving 37

decoding strategy 2

Grammar-constrained decoding used
Best-of-N scaffolding evaluated

reasoning control 2

Chain-of-thought reasoning core
Configurable reasoning effort used

inference quantization 3

MXFP4 weight quantization used
BF16 inference used
MXFP4 tensor packing used

serving parallelism 1

Tensor parallelism for MoE layers used

inference kernel 3

CUDA Graph used
Expert-optimized Triton kernels used
Optimized Triton MoE kernel with MXFP4 support used

context management 1

Excluding prior thinking from conversation history used

agentic scaffolding 25

Harmony format core
Role-based instruction hierarchy core
Assistant output channels core
Chain-of-thought in the analysis channel core
Developer message format core
Function-calling format core
Harmony channel annotations core
Harmony tool-call message format core
Python tool used
Browser tool used
Browsing tool with domain filtering used
Commentary-channel preambles used
Developer-defined function schemas used
Harmony history stop-token normalization used
Interleaving tool calls with chain-of-thought used
JSON Schema response formats used
Retaining chain-of-thought across tool-call turns used
Scrollable browser text window used
Stateful Python tool used
Stateless Python tool reference implementation used
System message format used
Tool output message format used
Tools section in the system message used
TypeScript-like function schema syntax used
StreamableParser optional

evaluation 9

filed at the root 2

ProtocolQA robustness validation used
Refusal behavior quantification used

benchmark 4

Capture-the-Flag (CTF) evaluation used
PaperBench used
SWE-bench Verified used
Three-configuration cyber range testing used

judge 1

LLM autograding with expert validation used

human & real-world evaluation 2

Cyber range exercises used
External expert review of safety evaluation methodology used

other 2

filed at the root 2

Request Caching in the Browser Tool used
Closed-Model Performance with Refusal Substitution not used

unfiled 14

Agentic capabilities core
harmony chat format core
Harmony output channels (final/analysis/commentary) core
OpenAI Harmony response format core
Attention memory-cost optimization in the Triton implementation used
Cross-provider failover routing for inference requests used
Fine-tuning hyperparameter verification used
Inference-time scaling plots for evaluation reporting used
no direct optimization pressure on chain-of-thought used
Python (stateful Jupyter) tool training used
PyTorch expandable segments allocator (PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True) for loading checkpoints used
Text-only multimodal benchmark alignment for comparability used
OpenAI Harmony renderer (openai_harmony) that renders messages into tokens optional
Sampling with temperature=1.0 and top_p=1.0 mentioned

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated document list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

DocumentKindPublisherStatus
openai.com/index/introducing-gpt-ossofficial blogOpenAInot fetched
arxiv.org/abs/2508.10925technical reportOpenAIread
huggingface.co/openai/gpt-oss-120bmodel cardOpenAIread
github.com/openai/gpt-osscode repoOpenAIread
cookbook.openai.com/articles/openai-harmonyvendor docsOpenAIread
openrouter.ai/openai/gpt-oss-120bvendor docsOpenRouterread
simonwillison.net/2025/Aug/5/gpt-ossthird party analysisSimon Willisonread