specific method · filed under model architecture
Single-Pass mHC
An mHC variant that shifts input-mixing coefficients by one block so each block consumes coefficients produced by the previous block.
Also called Single-PassmHC residual-stream mixing, Single-PassmHC.
- sources
- 2
- model
- 1
- lab adopt it
- 1
- strongest
- core
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Evidence
3 spans quoted from the sources, strongest treatment first.
Single-PassmHC revises residual-stream mixing to enable more efficient kernel fusion
Additional architectural components include Single-Pass mHC (revised residual-stream mixing with an efficient Mega-mHC kernel)
We therefore introduce Single-Pass mHC, which shifts the input-mixing coefficients by one block, i.e. every block consumes the mixing coefficients produced by the previous one
Filed alongside
Other methods under model architecture :: normalization & residual.