Open Models Almanac
Methodsoptimizationtraining precision

single mention · optimization

FP32 attention-output retention

FP32 attention-output retention keeps the attention output in FP32 during training.1 This method is adopted to correct the biased rounding error that arises in flash attention.2

Trade-off This choice doubles the on-chip footprint of the output tile.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. keep the attention output in FP32 during training — k3 tech report
  2. To correct the biased rounding error that arises in flash attention — k3 tech report
  3. This choice doubles the on-chip footprint of the output tile — k3 tech report

Filed under optimization › training precision