single mention · optimization
FP32 attention-output retention
FP32 attention-output retention keeps the attention output in FP32 during training.1 This method is adopted to correct the biased rounding error that arises in flash attention.2
Trade-off This choice doubles the on-chip footprint of the output tile.3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
keep the attention output in FP32 during training
— k3 tech reportTo correct the biased rounding error that arises in flash attention
— k3 tech reportThis choice doubles the on-chip footprint of the output tile
— k3 tech report
Filed under optimization › training precision