OpenAI
gpt-oss-120b
What gpt-oss-120b’s own documents say it is built from — every method with the sentence that describes it.
- Parameters
- 117B / 5.1B active
- Context
- 128K
- Attention
- sliding/global hybridalternating sliding-window (128) and full GQA layers, learned attention sinks
- Experts
- 128 routed, top-4, no shared
- Input
- text
- Licence
- Apache 2.0
- Released
- 2025-08-05
Against the consensus recipe
Its documents state 3 of the 18 methods the field agrees on.
Data curation
- Benchmark decontaminationnot in its documents
- Synthetic data generationnot in its documents
- Corpus deduplicationnot in its documents
Training objective
- Multi-Token Predictionnot in its documents
- Knowledge insulationnot in its documents
- Quantization-aware distillationnot in its documents
Model architecture
- Mixture of expertscore
- Multi-Token Predictionnot in its documents
- Grouped-query attentionused
Optimization
- Muonnot in its documents
- Expert parallelismnot in its documents
- Quantization-Aware Trainingnot in its documents
Post-training
- Supervised fine-tuningnot in its documents
- Asynchronous reinforcement learningnot in its documents
- Multi-Teacher On-Policy Distillationnot in its documents
Inference & serving
- Thinking modeused
- Multi-Token Predictionnot in its documents
- FP8not in its documents
Checked in its code
Its code shows 7 of the 19 architecture features checked — 2 of them are not stated in its documents.
Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.
Read from Hugging Face Transformers (modeling_gpt_oss.py, configuration_gpt_oss.py) and config.json at revision b5c939de8f, by anthropic/claude-sonnet-5 on 2026-11-07.
| Feature | Code | Its documents | Verdict |
|---|---|---|---|
| Grouped-query attention | in its code | adopted | 64 query heads share 8 key/value heads via repeat_kv in every attention layer show evidencenum_attention_heads = 64num_key_value_heads = 8 |
| Multi-head latent attention | not in its code | not stated | No low-rank KV latent projection exists; attention uses standard separate Q/K/V projections sized by head counts |
| Sliding-window attention code only | in its code | not stated | Layers tagged sliding_attention use sliding_window=128 as the local window size show evidencesliding_window = 128layer_types = ["sliding_attention", "full_attention", "sliding_attention", "full_attention"… |
| Interleaved sliding-window and global attention | in its code | adopted | layer_types alternates sliding_attention and full_attention across the 36 layers, each built with its own causal mask show evidencelayer_types = ["sliding_attention", "full_attention", "sliding_attention", "full_attention"… |
| Indexer-selected sparse attention (DeepSeek Sparse Attention) | not in its code | not stated | No indexer module or top-k token selection mechanism is present in this attention implementation |
| Linear-attention or state-space layers alongside full attention | not in its code | not stated | All layers use the same GptOssAttention softmax attention class; no linear-attention or SSM mixer layers exist |
| Gated DeltaNet layers | not in its code | not stated | No gated delta-rule recurrent layer implementation is present in this file |
| Mamba-2 layers | not in its code | not stated | No Mamba-2 SSM block implementation exists in this modeling file |
| Learnable attention sink code only | in its code | not stated | Each attention layer has a learned per-head sink parameter concatenated into the softmax denominator show evidence |
| Gated attention output | not in its code | not stated | No sigmoid gate is applied to the attention output before the output projection; attn_output is reshaped and passed directly to o_proj show evidence |
| QK normalization | not in its code | not stated | Query and key states are projected and rotary-embedded but never normalized before the attention dot product show evidence |
| Partial RoPE | not in its code | not stated | Rotary embedding is applied via chunking the full head dimension in half (first_half/second_half), covering the entire head_dim rather… show evidenceRotary embedding is applied via chunking the full head dimension in half (first_half/second_half), covering the entire head_dim rather than a partial slice |
| YaRN RoPE scaling | in its code | adopted | rope_scaling specifies rope_type yarn with factor 32.0, and the rotary embedding module dispatches to the YaRN init function show evidencerope_scaling.rope_type = "yarn"rope_scaling.factor = 32.0 |
| Layers without positional encoding (NoPE) | not in its code | not stated | All attention layers receive the same rotary position embeddings computed once and passed to every layer; there is no layer-conditional… show evidenceAll attention layers receive the same rotary position embeddings computed once and passed to every layer; there is no layer-conditional skip of RoPE |
| Mixture of experts | in its code | adopted | A learned router selects top-4 of 128 experts per token for the feed-forward computation show evidencenum_local_experts = 128num_experts_per_tok = 4 |
| Shared expert | not in its code | not stated | No shared-expert module exists in GptOssMLP; only routed experts process tokens, each token's output being purely a weighted sum of… show evidenceNo shared-expert module exists in GptOssMLP; only routed experts process tokens, each token's output being purely a weighted sum of selected experts |
| Auxiliary-loss-free load balancing (selection bias) | not in its code | not stated | The router computes top-k directly from raw logits with no added selection bias term, and no e_score_correction_bias or topk_method config… show evidenceThe router computes top-k directly from raw logits with no added selection bias term, and no e_score_correction_bias or topk_method config key is present |
| Multi-token prediction layers | not in its code | not stated | No multi-token-prediction modules or related config keys (e.g. num_nextn_predict_layers) appear in this config or code |
| SwiGLU feed-forward | in its code | adopted | Expert MLPs compute a sigmoid-gated (SiLU-style) gate times up-projection, with hidden_act silu, implementing a clamped SwiGLU show evidencehidden_act = "silu" |
token mixer 3
softmax attention 2
“Each attention head has a learned bias in the denominator of the softmax”
grouped-query attention 1
“uses Grouped Query Attention (GQA [12][13]) with 8 key-value heads”
hybrid layer stacking 1
“attention blocks alternate between banded window and fully dense patterns”
channel mixer 3
dense feed-forward network 1
“The MoE blocks use the gated SwiGLU [9] activation function”
mixture of experts 2
“117B-parameter Mixture-of-Experts (MoE) language model ... activates 5.1B parameters per forward pass”
expert routing 1
“For both models, we select the top- experts for each token given by the router, and weight the output of each expert by the softmax of the…”
positional encoding 2
“extend the context length of dense layers to tokens using YaRN”
“We apply rotary position embeddings [14]”
quantization-aware training 1
“The models were post-trained with MXFP4 quantization of the MoE weights”
data filtering 1
“we filtered the data for harmful content in pre-training, especially around hazardous biosecurity knowledge, by reusing the CBRN…”
data mixture & curriculum 1
“We train the models on a text-only dataset with trillions of tokens, with a focus on STEM, coding, and general knowledge.”
tokenization 2
“Across all training stages, we utilize our o200k_harmony tokenizer”
“They are part of a new token vocabulary called o200k_harmony, which landed in OpenAI’s tiktoken tokenizer library this morning.”
1 more mentioned or evaluated
data mixture & curriculum “Tested analogous datasets during RL training to confirm no harmful uplift”
supervised fine-tuning 3
“we created adversarially fine-tuned versions of gpt-oss-120b for the two categories in which we believed there was a plausible chance that…”
“We collected examples of these different roles of messages conflicting with each other, and supervised gpt-oss to follow the instructions…”
“Fine-tunable: Fully customize models to your specific use case through parameter fine-tuning.”
reinforcement learning algorithm 3
“After pre-training, we post-train the models using similar CoT RL techniques as OpenAI o3.”
“Helpful-only training: We performed an additional stage of reinforcement learning to reward answers that comply with unsafe prompts.”
“We focus on incremental reinforcement learning, which we believe is the most apt technical approach.”
preference optimization 1
“During post-training, we use deliberative alignment[29] to teach the models to refuse on a wide range of content”
agentic post-training 2
“During post-training, we also teach the models to use different agentic tools:”
“The model was trained to use a python tool to perform calculations and other actions as part of its chain-of-thought.”
decoding strategy 1
“This prompt alone will, however, only influence the model’s behavior but doesn’t guarantee the full adherence to the schema. For this you…”
reasoning control 2
“Full chain-of-thought: Gain complete access to the model’s reasoning process, facilitating easier debugging and increased trust in outputs.”
“We train the models to support three reasoning levels: low, medium, and high.”
inference quantization 3
“is optimized to run on a single H100 GPU with native MXFP4 quantization”
“In this implementation, we upcast all weights to BF16 and run the model in BF16.”
“We pack every two values in one `uint8` value.”
serving parallelism 1
“with a small addition of supporting tensor parallelism in MoE so that the larger model can run with this code (e.g., on 4xH100 or 2xH200)”
inference kernel 3
“incl. using CUDA graphs and basic caching”
“The gpt-oss models trained on NVIDIA H100 GPUs using the PyTorch framework with expert-optimized Triton kernels.”
“optimized triton MoE kernel”
context management 1
“In general, you should drop any previous CoT content on subsequent sampling if the responses by the assistant ended in a message to the…”
agentic scaffolding 22
“Both models were trained on our harmony response format and should only be used with the harmony format as it will not work correctly…”
“These roles also represent the information hierarchy that the model applies in case there are any instruction conflicts: system >…”
“Assistant messages can be output in three different “channels”. These are being used to separate between user-facing responses and…”
“The model will output its raw chain-of-thought (CoT) as assistant messages into the analysis channel while the final response will be…”
“The developer message represents what is commonly considered the “system prompt”. It contains the instructions that are provided to the…”
“All functions that are available to the model should be defined in the developer message in a dedicated Tools section.”
“If the model decides to call a tool it will define a recipient in the header of the message using the format to={name}.”
“The cybermax model was also trained to use a browsing tool with a domain block that filters out any websites containing eval writeups or…”
“These tools should be defined in the system message not in the developer message by adding a # Tools section.”
“If this is the case it will generate an assistant message on the commentary channel that, unlike the chain-of-thought, is intended to be…”
“Arbitrary developer functions, where one can specify function schemas in a Developer message similar to the OpenAI API.”
“When you add the assistant’s generated reply to conversation history for the next turn, replace the trailing <|return|> with <|end|> so…”
“We use the system message to define: - The identity of the model — This should always stay as You are ChatGPT, a large language model…”
“This format enables gpt-oss to provide advanced agentic features including interleaving tool calls within the CoT or providing preambles…”
“To control the output behavior of the model, you can define a response format at the end of the developer message with the following…”
“Use this tool to execute Python code in your chain of thought.”
“The exception for this is tool/function calling. The model is able to call tools as part of its chain-of-thought and because of that, we…”
“During the training the model used a stateful tool which makes running tools between CoT loops easier.”
“This reference implementation, however, uses a stateless mode.”
“A tool message has the following format: <|start|>{toolname} to=assistant<|channel|>commentary<|message|>{output}<|end|>”
“To define the functions we use a TypeScript-like type syntax and wrap the functions into a dedicated functions namespace.”
“Additionally the openai_harmony library also includes a StreamableParser for parsing and decoding as the model is generating new tokens.”
1 more mentioned or evaluated
decoding strategy “Tested internal 'Best of K' scaffolding in cyber evaluations.”
filed at the root 2
“We validated our protocol troubleshooting results by checking that the model never refused, adding more protocol-debugging training data…”
“Explicitly separated refusal-based failures from other failure modes and reported pre- and post-naughtification rates.”
benchmark 4
“To evaluate the model against the CTFs we give it access to a headlessLinux distribution with common offensive cybersecurity tools…”
“For objective evaluation, we develop rubrics that hierarchically decompose each replication task into smaller sub-tasks with clear grading…”
“All SWE-bench evaluation runs use a fixed subset of n=477 verified tasks which have been validated on our internal infrastructure.”
“Each scenario is tested under three configurations: Normal, where the model is given the goal, SSH key, and nothing more; With Hints…”
judge 1
“We used the OpenAI o1-preview (pre-mitigation) model as an autograder, validating agreement with a trusted biosecurity expert.”
human & real-world evaluation 2
“Cyber range exercises measure a model's ability to conduct fully end-to-end cyber operations in a realistic, emulated network.”
“We engaged a small group of external safety experts (METR, SecureBio, and Daniel Kang) to independently review and validate our malicious…”
other 2
filed at the root 1
“To improve performance the tool caches requests so that the model can revisit a different part of a page without having to reload the page.”
Not used 1
“Recommendation: Compute combined performance using closed models where non-refusal responses are substituted, treating refusals as zero.”
unfiled 14
“Agentic capabilities: Use the models’ native capabilities for function calling, web browsing, Python code execution, and Structured Outputs.”
“For the models’ training, we use a custom chat format known as the harmony chat format.”
“Three different channels for output: final, analysis and commentary.”
“openai/harmony is a brand new open source project from OpenAI (again, Apache 2) which implements a new response format that was created…”
“It also has some optimization on the attention code to reduce the memory cost.”
“When an error occurs in an upstream provider, we can recover by routing to another healthy provider”
“Aligned with internal researchers on best hyperparameter settings for maximum performance and changed when necessary.”
“Added plots for both bio and cyber evals showing how performance scales with number of trials.”
“we decided not to put any direct optimization pressure on the CoT for either of our two open-weight models.”
“A python tool, which allows the model to run code in a stateful Jupyter notebook environment.”
“make sure to turn on the expandable allocator to avoid crashes when loading weights from the checkpoint.”
“Ran text-only versions of Multimodal Virology Troubleshooting and updated results to improve comparability.”
“We recommend using our harmony renderer through PyPI or crates.io when possible as it will automatically handle rendering your messages in…”
1 more mentioned or evaluated
“We recommend sampling with `temperature=1.0` and `top_p=1.0`.”
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- gpt-oss-120b architecture card — LLM Architecture Gallery explainer sebastianraschka.com
Sources
The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.
| Source | Kind | Publisher | Status |
|---|---|---|---|
| openai.com/index/introducing-gpt-oss | official blog | OpenAI | not fetched |
| arxiv.org/abs/2508.10925 | technical report | OpenAI | read |
| huggingface.co/openai/gpt-oss-120b | model card | OpenAI | read |
| github.com/openai/gpt-oss | code repo | OpenAI | read |
| cookbook.openai.com/articles/openai-harmony | vendor docs | OpenAI | read |
| openrouter.ai/openai/gpt-oss-120b | vendor docs | OpenRouter | read |
| simonwillison.net/2025/Aug/5/gpt-oss | third party analysis | Simon Willison | read |
| recipes.vllm.ai/openai/gpt-oss-120b | vendor docs | vLLM | not fetched |