Laguna-S-2.1
PoolsideWe use BF16 mixed precision training throughout all stages with the master weights in FP32
specific method · filed under optimization
Mixed-precision training using bfloat16 for most operations, with some evidence specifying FP32 master weights or retaining sensitive operations in full precision.
The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See training precision for the methods filed beside it.
Also called mixed-precision training using BF16, bfloat16 mixed-precision training, bfloat16 precision, Mixed-precision training with bfloat16.
5 quotes from 5 sources, one card per model, each opening with its strongest.
We use BF16 mixed precision training throughout all stages with the master weights in FP32
We use automatic mixed precision with bfloat16 for most operations, while keeping numerically sensitive operations such as layer normalization and RoPE in full precision.
we enable Automatic Mixed Precision (AMP) with bfloat16 for most operations.
mixed-precision training using BF16 is enabled throughout to improve computational efficiency and reduce memory consumption
we leverage bfloat16 precision and torch.compile()