Open Models Almanac
Modelsgpt-oss-120b

OpenAI

gpt-oss-120b

What gpt-oss-120b’s own documents say it is built from — every method with the sentence that describes it.

Parameters
117B / 5.1B active
Context
128K
Attention
sliding/global hybridalternating sliding-window (128) and full GQA layers, learned attention sinks
Experts
128 routed, top-4, no shared
Input
text
Licence
Apache 2.0
Released
2025-08-05

Against the consensus recipe

Its documents state 3 of the 18 methods the field agrees on.

The recipe from the overview: the methods the most labs adopt, per stage. A missing one is something its documents do not say, not something the model lacks.
  1. Data curation

  2. Training objective

  3. Model architecture

  4. Optimization

  5. Post-training

  6. Inference & serving

Checked in its code

Its code shows 7 of the 19 architecture features checked — 2 of them are not stated in its documents.

Sources say what a lab chose to describe; the modeling code says what the checkpoint runs. Each feature below was put to a code verifier, and only the config values and source lines it cited that were found in the files are shown, linked to the line.

Read from Hugging Face Transformers (modeling_gpt_oss.py, configuration_gpt_oss.py) and config.json at revision b5c939de8f, by anthropic/claude-sonnet-5 on 2026-11-07.

FeatureCodeIts documentsVerdict
Grouped-query attention in its codeadopted

64 query heads share 8 key/value heads via repeat_kv in every attention layer

show evidencenum_attention_heads = 64num_key_value_heads = 8
self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
modeling_gpt_oss.py · L268
key_states = repeat_kv(key, module.num_key_value_groups)
    value_states = repeat_kv(value, module.num_key_value_groups)
modeling_gpt_oss.py · L237–238
Multi-head latent attention not in its codenot stated

No low-rank KV latent projection exists; attention uses standard separate Q/K/V projections sized by head counts

Sliding-window attention code only in its codenot stated

Layers tagged sliding_attention use sliding_window=128 as the local window size

show evidencesliding_window = 128layer_types = ["sliding_attention", "full_attention", "sliding_attention", "full_attention"…
self.sliding_window = config.sliding_window if self.layer_type == "sliding_attention" else None
modeling_gpt_oss.py · L284
Interleaved sliding-window and global attention in its codeadopted

layer_types alternates sliding_attention and full_attention across the 36 layers, each built with its own causal mask

show evidencelayer_types = ["sliding_attention", "full_attention", "sliding_attention", "full_attention"…
causal_mask_mapping = {
                "full_attention": create_causal_mask(**mask_kwargs),
                "sliding_attention": create_sliding_window_causal_mask(**mask_kwargs),
            }
modeling_gpt_oss.py · L466–469
Indexer-selected sparse attention (DeepSeek Sparse Attention) not in its codenot stated

No indexer module or top-k token selection mechanism is present in this attention implementation

Linear-attention or state-space layers alongside full attention not in its codenot stated

All layers use the same GptOssAttention softmax attention class; no linear-attention or SSM mixer layers exist

Gated DeltaNet layers not in its codenot stated

No gated delta-rule recurrent layer implementation is present in this file

Mamba-2 layers not in its codenot stated

No Mamba-2 SSM block implementation exists in this modeling file

Learnable attention sink code only in its codenot stated

Each attention layer has a learned per-head sink parameter concatenated into the softmax denominator

show evidence
self.sinks = nn.Parameter(torch.empty(config.num_attention_heads))
modeling_gpt_oss.py · L285
sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
    combined_logits = torch.cat([attn_weights, sinks], dim=-1)
modeling_gpt_oss.py · L243–244
Gated attention output not in its codenot stated

No sigmoid gate is applied to the attention output before the output projection; attn_output is reshaped and passed directly to o_proj

show evidence
attn_output = attn_output.reshape(*input_shape, -1).contiguous()
        attn_output = self.o_proj(attn_output)
modeling_gpt_oss.py · L325–326
QK normalization not in its codenot stated

Query and key states are projected and rotary-embedded but never normalized before the attention dot product

show evidence
query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
modeling_gpt_oss.py · L298–300
Partial RoPE not in its codenot stated

Rotary embedding is applied via chunking the full head dimension in half (first_half/second_half), covering the entire head_dim rather…

show evidence

Rotary embedding is applied via chunking the full head dimension in half (first_half/second_half), covering the entire head_dim rather than a partial slice

first_half, second_half = torch.chunk(x, 2, dim=-1)
    first_ = first_half * cos - second_half * sin
    second_ = second_half * cos + first_half * sin
    return torch.cat((first_, second_), dim=-1)
modeling_gpt_oss.py · L212–215
YaRN RoPE scaling in its codeadopted

rope_scaling specifies rope_type yarn with factor 32.0, and the rotary embedding module dispatches to the YaRN init function

show evidencerope_scaling.rope_type = "yarn"rope_scaling.factor = 32.0
self.rope_type = self.config.rope_parameters["rope_type"]
        rope_init_fn: Callable = self.compute_default_rope_parameters
        if self.rope_type != "default":
            rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
modeling_gpt_oss.py · L156–159
Layers without positional encoding (NoPE) not in its codenot stated

All attention layers receive the same rotary position embeddings computed once and passed to every layer; there is no layer-conditional…

show evidence

All attention layers receive the same rotary position embeddings computed once and passed to every layer; there is no layer-conditional skip of RoPE

cos, sin = position_embeddings
        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
modeling_gpt_oss.py · L302–303
Mixture of experts in its codeadopted

A learned router selects top-4 of 128 experts per token for the feed-forward computation

show evidencenum_local_experts = 128num_experts_per_tok = 4
router_top_value, router_indices = torch.topk(router_logits, self.top_k, dim=-1)  # (num_tokens, top_k)
modeling_gpt_oss.py · L127
Shared expert not in its codenot stated

No shared-expert module exists in GptOssMLP; only routed experts process tokens, each token's output being purely a weighted sum of…

show evidence

No shared-expert module exists in GptOssMLP; only routed experts process tokens, each token's output being purely a weighted sum of selected experts

self.router = GptOssTopKRouter(config)
        self.experts = GptOssExperts(config)
modeling_gpt_oss.py · L136–137
Auxiliary-loss-free load balancing (selection bias) not in its codenot stated

The router computes top-k directly from raw logits with no added selection bias term, and no e_score_correction_bias or topk_method config…

show evidence

The router computes top-k directly from raw logits with no added selection bias term, and no e_score_correction_bias or topk_method config key is present

router_logits = F.linear(hidden_states, self.weight, self.bias)  # (num_tokens, num_experts)
        router_top_value, router_indices = torch.topk(router_logits, self.top_k, dim=-1)  # (num_tokens, top_k)
modeling_gpt_oss.py · L126–127
Multi-token prediction layers not in its codenot stated

No multi-token-prediction modules or related config keys (e.g. num_nextn_predict_layers) appear in this config or code

SwiGLU feed-forward in its codeadopted

Expert MLPs compute a sigmoid-gated (SiLU-style) gate times up-projection, with hidden_act silu, implementing a clamped SwiGLU

show evidencehidden_act = "silu"
glu = gate * torch.sigmoid(gate * self.alpha)
        gated_output = (up + 1) * glu
modeling_gpt_oss.py · L85–86

token mixer 3

softmax attention 2

Off-by-one attention used

“Each attention head has a learned bias in the denominator of the softmax”

grouped-query attention 1

Grouped-query attention used

“uses Grouped Query Attention (GQA [12][13]) with 8 key-value heads”

hybrid layer stacking 1

Hybrid Sliding-Window and Global Attention used

“attention blocks alternate between banded window and fully dense patterns”

channel mixer 3

dense feed-forward network 1

SwiGLU used

“The MoE blocks use the gated SwiGLU [9] activation function”

mixture of experts 2

Mixture of experts core

“117B-parameter Mixture-of-Experts (MoE) language model ... activates 5.1B parameters per forward pass”

expert routing 1

Top-k routing with softmax over selected experts core

“For both models, we select the top- experts for each token given by the router, and weight the output of each expert by the softmax of the…”

positional encoding 2

YaRN used

“extend the context length of dense layers to tokens using YaRN”

Rotary Position Embedding used

“We apply rotary position embeddings [14]”

normalization & residual 2

RMSNorm used

“applying root mean square normalization [6] on the activations before each attention and MoE block”

Pre-LN used

“Similar to GPT-2 we use Pre-LN placement [7][8].”

quantization-aware training 1

MXFP4 Quantization default

“The models were post-trained with MXFP4 quantization of the MoE weights”

data filtering 1

CBRN safety filtering of pre-training data used

“we filtered the data for harmful content in pre-training, especially around hazardous biosecurity knowledge, by reusing the CBRN…”

data mixture & curriculum 1

Text-only pre-training corpus focused on STEM, coding, and general knowledge used

“We train the models on a text-only dataset with trillions of tokens, with a focus on STEM, coding, and general knowledge.”

tokenization 2

o200k_harmony tokenizer used

“Across all training stages, we utilize our o200k_harmony tokenizer”

o200k_harmony vocabulary used

“They are part of a new token vocabulary called o200k_harmony, which landed in OpenAI’s tiktoken tokenizer library this morning.”

1 more mentioned or evaluated
ProtocolQA-aligned RL training datasets evaluated

data mixture & curriculum “Tested analogous datasets during RL training to confirm no harmful uplift”

supervised fine-tuning 3

Adversarial fine-tuning used

“we created adversarially fine-tuned versions of gpt-oss-120b for the two categories in which we believed there was a plausible chance that…”

Instruction hierarchy training used

“We collected examples of these different roles of messages conflicting with each other, and supervised gpt-oss to follow the instructions…”

Fine-tuning optional

“Fine-tunable: Fully customize models to your specific use case through parameter fine-tuning.”

reinforcement learning algorithm 3

Chain-of-thought reinforcement learning core

“After pre-training, we post-train the models using similar CoT RL techniques as OpenAI o3.”

Helpful-only adversarial reinforcement learning used

“Helpful-only training: We performed an additional stage of reinforcement learning to reward answers that comply with unsafe prompts.”

Incremental reinforcement learning used

“We focus on incremental reinforcement learning, which we believe is the most apt technical approach.”

preference optimization 1

Deliberative alignment used

“During post-training, we use deliberative alignment[29] to teach the models to refuse on a wide range of content”

agentic post-training 2

Agentic tool-use post-training used

“During post-training, we also teach the models to use different agentic tools:”

Python-tool use during chain-of-thought used

“The model was trained to use a python tool to perform calculations and other actions as part of its chain-of-thought.”

decoding strategy 1

Grammar-constrained decoding used

“This prompt alone will, however, only influence the model’s behavior but doesn’t guarantee the full adherence to the schema. For this you…”

reasoning control 2

Full chain-of-thought output core

“Full chain-of-thought: Gain complete access to the model’s reasoning process, facilitating easier debugging and increased trust in outputs.”

Thinking mode used

“We train the models to support three reasoning levels: low, medium, and high.”

inference quantization 3

MXFP4 weight quantization used

“is optimized to run on a single H100 GPU with native MXFP4 quantization”

BF16 inference used

“In this implementation, we upcast all weights to BF16 and run the model in BF16.”

MXFP4 block-scaled tensor packing used

“We pack every two values in one `uint8` value.”

serving parallelism 1

Tensor parallelism for MoE layers used

“with a small addition of supporting tensor parallelism in MoE so that the larger model can run with this code (e.g., on 4xH100 or 2xH200)”

inference kernel 3

CUDA Graph capture and replay used

“incl. using CUDA graphs and basic caching”

Expert-optimized Triton kernels used

“The gpt-oss models trained on NVIDIA H100 GPUs using the PyTorch framework with expert-optimized Triton kernels.”

Optimized Triton MoE kernel with MXFP4 support used

“optimized triton MoE kernel”

context management 1

Excluding prior thinking from multi-turn history used

“In general, you should drop any previous CoT content on subsequent sampling if the responses by the assistant ended in a message to the…”

agentic scaffolding 22

Harmony format core

“Both models were trained on our harmony response format and should only be used with the harmony format as it will not work correctly…”

Role-based instruction hierarchy core

“These roles also represent the information hierarchy that the model applies in case there are any instruction conflicts: system >…”

Assistant output channels core

“Assistant messages can be output in three different “channels”. These are being used to separate between user-facing responses and…”

Chain-of-thought in the analysis channel core

“The model will output its raw chain-of-thought (CoT) as assistant messages into the analysis channel while the final response will be…”

Developer message format core

“The developer message represents what is commonly considered the “system prompt”. It contains the instructions that are provided to the…”

Harmony function-calling format core

“All functions that are available to the model should be defined in the developer message in a dedicated Tools section.”

Harmony tool-call message format core

“If the model decides to call a tool it will define a recipient in the header of the message using the format to={name}.”

Browser tool used

“The cybermax model was also trained to use a browsing tool with a domain block that filters out any websites containing eval writeups or…”

Built-in tool declaration in the system message used

“These tools should be defined in the system message not in the developer message by adding a # Tools section.”

Commentary-channel preambles used

“If this is the case it will generate an assistant message on the commentary channel that, unlike the chain-of-thought, is intended to be…”

Developer-defined function schemas used

“Arbitrary developer functions, where one can specify function schemas in a Developer message similar to the OpenAI API.”

Harmony history end-token normalization used

“When you add the assistant’s generated reply to conversation history for the next turn, replace the trailing <|return|> with <|end|> so…”

Harmony system message format used

“We use the system message to define: - The identity of the model — This should always stay as You are ChatGPT, a large language model…”

Interleaved tool calls in chain-of-thought used

“This format enables gpt-oss to provide advanced agentic features including interleaving tool calls within the CoT or providing preambles…”

JSON Schema response formats used

“To control the output behavior of the model, you can define a response format at the end of the developer message with the following…”

Python execution tool in chain-of-thought used

“Use this tool to execute Python code in your chain of thought.”

Retaining chain-of-thought across tool calls used

“The exception for this is tool/function calling. The model is able to call tools as part of its chain-of-thought and because of that, we…”

Stateful Python tool used

“During the training the model used a stateful tool which makes running tools between CoT loops easier.”

Stateless Python tool reference implementation used

“This reference implementation, however, uses a stateless mode.”

Tool message format used

“A tool message has the following format: <|start|>{toolname} to=assistant<|channel|>commentary<|message|>{output}<|end|>”

TypeScript-like function schema syntax used

“To define the functions we use a TypeScript-like type syntax and wrap the functions into a dedicated functions namespace.”

StreamableParser optional

“Additionally the openai_harmony library also includes a StreamableParser for parsing and decoding as the model is generating new tokens.”

1 more mentioned or evaluated
Best-of-N selection evaluated

decoding strategy “Tested internal 'Best of K' scaffolding in cyber evaluations.”

filed at the root 2

ProtocolQA robustness validation used

“We validated our protocol troubleshooting results by checking that the model never refused, adding more protocol-debugging training data…”

Refusal behavior quantification used

“Explicitly separated refusal-based failures from other failure modes and reported pre- and post-naughtification rates.”

benchmark 4

Capture-the-Flag (CTF) evaluation used

“To evaluate the model against the CTFs we give it access to a headlessLinux distribution with common offensive cybersecurity tools…”

PaperBench hierarchical rubric evaluation used

“For objective evaluation, we develop rubrics that hierarchically decompose each replication task into smaller sub-tasks with clear grading…”

SWE-bench Verified evaluation used

“All SWE-bench evaluation runs use a fixed subset of n=477 verified tasks which have been validated on our internal infrastructure.”

Three-configuration cyber range testing used

“Each scenario is tested under three configurations: Normal, where the model is given the goal, SSH key, and nothing more; With Hints…”

judge 1

LLM autograding with expert validation used

“We used the OpenAI o1-preview (pre-mitigation) model as an autograder, validating agreement with a trusted biosecurity expert.”

human & real-world evaluation 2

Cyber range exercises used

“Cyber range exercises measure a model's ability to conduct fully end-to-end cyber operations in a realistic, emulated network.”

External expert review of safety evaluation methodology used

“We engaged a small group of external safety experts (METR, SecureBio, and Daniel Kang) to independently review and validate our malicious…”

filed at the root 1

Browser-tool request caching used

“To improve performance the tool caches requests so that the model can revisit a different part of a page without having to reload the page.”

Not used 1

Combined performance with refusal substitution not used

“Recommendation: Compute combined performance using closed models where non-refusal responses are substituted, treating refusals as zero.”

unfiled 14

Agentic capabilities core

“Agentic capabilities: Use the models’ native capabilities for function calling, web browsing, Python code execution, and Structured Outputs.”

harmony chat format core

“For the models’ training, we use a custom chat format known as the harmony chat format.”

Harmony output channels (final/analysis/commentary) core

“Three different channels for output: final, analysis and commentary.”

OpenAI Harmony response format core

“openai/harmony is a brand new open source project from OpenAI (again, Apache 2) which implements a new response format that was created…”

Attention memory-cost optimization in the Triton implementation used

“It also has some optimization on the attention code to reduce the memory cost.”

Cross-provider failover routing for inference requests used

“When an error occurs in an upstream provider, we can recover by routing to another healthy provider”

Fine-tuning hyperparameter verification used

“Aligned with internal researchers on best hyperparameter settings for maximum performance and changed when necessary.”

Inference-time scaling plots for evaluation reporting used

“Added plots for both bio and cyber evals showing how performance scales with number of trials.”

no direct optimization pressure on chain-of-thought used

“we decided not to put any direct optimization pressure on the CoT for either of our two open-weight models.”

Python (stateful Jupyter) tool training used

“A python tool, which allows the model to run code in a stateful Jupyter notebook environment.”

PyTorch expandable segments allocator (PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True) for loading checkpoints used

“make sure to turn on the expandable allocator to avoid crashes when loading weights from the checkpoint.”

Text-only multimodal benchmark alignment for comparability used

“Ran text-only versions of Multimodal Virology Troubleshooting and updated results to improve comparability.”

OpenAI Harmony renderer (openai_harmony) that renders messages into tokens optional

“We recommend using our harmony renderer through PyPI or crates.io when possible as it will automatically handle rendering your messages in…”

1 more mentioned or evaluated
Sampling with temperature=1.0 and top_p=1.0 mentioned

“We recommend sampling with `temperature=1.0` and `top_p=1.0`.”

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

Sources

The curated source list for this model. "not fetched" means the URL is recorded but its text was not read in the current run.

SourceKindPublisherStatus
openai.com/index/introducing-gpt-ossofficial blogOpenAInot fetched
arxiv.org/abs/2508.10925technical reportOpenAIread
huggingface.co/openai/gpt-oss-120bmodel cardOpenAIread
github.com/openai/gpt-osscode repoOpenAIread
cookbook.openai.com/articles/openai-harmonyvendor docsOpenAIread
openrouter.ai/openai/gpt-oss-120bvendor docsOpenRouterread
simonwillison.net/2025/Aug/5/gpt-ossthird party analysisSimon Willisonread
recipes.vllm.ai/openai/gpt-oss-120bvendor docsvLLMnot fetched
↑