single mention · model architecture
Residual-stream width selection
Residual-stream width selection is the choice of the residual width dmodel, the dimensionality of the shared representation read and updated by both attention and MoE blocks.1 Configurations are compared at matched training FLOPs, where a wider residual stream comes with fewer attention heads and narrower experts.2
Trade-off A wider residual stream improves training loss and accuracy but raises parameter and compute cost, memory and communication volume.3,4
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
The residual width dmodel determines the dimensionality of the shared representation read and updated by both attention and MoE blocks
— tech reportUnder approximately matched training FLOPs, a wider residual stream therefore comes with fewer attention heads of the same head dimension and with narrower experts
— tech reportIncreasing dmodel raises the parameter and compute cost of the associated projections, residual-activation memory and communication volume
— tech reportAfter learning-rate selection, training loss and mean accuracy improve with residual width
— tech report
Also called residual width.
Filed under model architecture