Open Models Almanac
Methodsmodel architecturenormalization & residual

specific method · filed under model architecture

Manifold-Constrained Hyper-Connections

mHC widens the residual stream into several parallel streams that each sublayer reads, writes and mixes with learned coefficients, and rescales the mixing matrix with Sinkhorn iterations so its rows and columns sum to one. A product of such matrices stays in the same set, so the learned mixing cannot amplify the signal however many layers it passes.

Written for the explainer Rethinking the residual stream.

Also called mHC.

sources
9
models
5
labs adopt it
2
strongest
core

Documented in

Evidence

9 quotes from 9 sources, one card per model, each opening with its strongest.

How sources treat it
evaluated 1used 5core 3

DeepSeek-V4-Pro

DeepSeekadopted4 sources

DeepSeek-V4 series incorporate Manifold-Constrained Hyper-Connections (mHC) to strengthen the conventional residual connections

coremodel architecturein DeepSeek-V4 seriesDeepSeek
3 more from this model

Manifold-Constrained Hyper-Connections (mHC): We incorporate mHC to strengthen conventional residual connections, enhancing stability of signal propagation across layers while preserving model expressivity.

usedmodel architecturein DeepSeek-V4 seriesDeepSeek

Manifold-Constrained Hyper-Connections (mHC): constrains residual mapping onto the manifold of doubly stochastic matrices (Birkhoff polytope), enhancing signal propagation stability while preserving model expressivity.

usedmodel architecturein DeepSeek-V4DeepSeek

mHC is confirmed in V4 as described in the December 2025 paper

usedmodel architecturein DeepSeek-V4DeepSeek

DeepSeek-V4-Flash

DeepSeekadopted3 sources

DeepSeek-V4 series incorporate Manifold-Constrained Hyper-Connections (mHC) to strengthen the conventional residual connections

coremodel architecturein DeepSeek-V4 seriesDeepSeek
2 more from this model

Manifold-Constrained Hyper-Connections (mHC): constrains residual mapping onto the manifold of doubly stochastic matrices (Birkhoff polytope), enhancing signal propagation stability while preserving model expressivity.

usedmodel architecturein DeepSeek-V4DeepSeek

incorporate mHC to strengthen conventional residual connections

usedmodel architecturein DeepSeek-V4-FlashDeepSeek

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑