Model techniques map
Taxonomymodel architecturenormalization & residual

taxonomy node · level 2

normalization & residual

28 methods filed at this node or below it, from the sources of 10 models.

model architecture :: normalization & residual

Matching aids for the classifier: RMSNorm; QK-Norm; QK-Clip; hyper-connections; attention residuals; AttnRes.

In this branch 28

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 28

Manifold-Constrained Hyper-Connections core · 8 sources · 8 quotes
Attention Residuals (AttnRes) core · 7 sources · 7 quotes
Gated Residual core · 6 sources · 8 quotes
Identity Hyper-Connections core · 3 sources · 3 quotes
Hyper-Connections core · 2 sources · 2 quotes
Per-Branch Scalar Write Gate core · 2 sources · 2 quotes
Single-Pass mHC core · 2 sources · 3 quotes
All-branch Elementwise Read Gate core · 1 source · 1 quote
Block AttnRes core · 1 source · 2 quotes
Dynamic Gating of Residual Reads and Writes core · 1 source · 1 quote
Elementwise Data-Dependent Residual Read Gate core · 1 source · 1 quote
Four-Branch Residual Stream core · 1 source · 1 quote
Four-Stream Hyper-Connection Combine Update core · 1 source · 1 quote
GatedNorm core · 1 source · 3 quotes
Independent Per-Branch Normalization core · 1 source · 2 quotes
Low-Rank Gated Mix core · 1 source · 1 quote
Two-Phase Block AttnRes Schedule core · 1 source · 1 quote
Widened Residual Stream core · 1 source · 1 quote
RMSNorm used · 3 sources · 3 quotes
Bounded Positive Gates used · 1 source · 1 quote
Data-Dependent Residual Read and Write Operators used · 1 source · 1 quote
Post-Embedding RMSNorm used · 1 source · 1 quote
Pre-LN used · 1 source · 1 quote
Zero-Centered RMSNorm used · 1 source · 1 quote
Simplified AltUp evaluated · 1 source · 1 quote
Full Attention Residuals mentioned · 1 source · 1 quote
QK-Clip not used · 2 sources · 2 quotes
Sparse Gated Residual Writes not used · 1 source · 1 quote

By model

Which of this branch's techniques each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modeltechniques
GLM-5.3-Flash coreManifold-Constrained Hyper-Connections core—
DeepSeek-V4.1-Flash coreSingle-Pass mHC coreRMSNorm used—
Hy4-preview coreIdentity Hyper-Connections core—
DeepSeek-V4-Flash coreManifold-Constrained Hyper-Connections coreRMSNorm usedQK-Clip not used—
DeepSeek-V4-Flash-Vision-Exp coreHyper-Connections core—
DeepSeek-V4-Pro coreManifold-Constrained Hyper-Connections coreRMSNorm usedQK-Clip not used—
Inkling usedPost-Embedding RMSNorm used—
Kimi K3 coreAttention Residuals (AttnRes) coreBlock AttnRes coreTwo-Phase Block AttnRes Schedule coreFull Attention Residuals mentioned—
Qwen3.8-Flash-Next coreAll-branch Elementwise Read Gate coreDynamic Gating of Residual Reads and Writes coreElementwise Data-Dependent Residual Read Gate coreFour-Branch Residual Stream coreFour-Stream Hyper-Connection Combine Update coreGated Residual coreGatedNorm coreIndependent Per-Branch Normalization coreLow-Rank Gated Mix corePer-Branch Scalar Write Gate coreWidened Residual Stream coreBounded Positive Gates usedData-Dependent Residual Read and Write Operators usedZero-Centered RMSNorm usedHyper-Connections evaluatedManifold-Constrained Hyper-Connections evaluatedSimplified AltUp evaluatedQK-Clip not usedSparse Gated Residual Writes not used—
gpt-oss-120b usedPre-LN usedRMSNorm used—