single mention · optimization
FP8 residual-state storage
Storing the widened residual state in FP8 to reduce bytes moved compared with BF16.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See training precision for the methods filed beside it.
Also called FP8 storage for the widened residual state, keep the residual state in FP8.
Filed under optimization › training precision