implementation detail · filed under optimization
BF16
The BF16 floating-point format is used for selected layers or precision-sensitive computations.
Also called BF16 precision, BF16 testing, keeping attention projections in BF16, BF16 (bfloat16).
- sources
- 5
- models
- 3
- labs adopt it
- 2
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- A Study of BFLOAT16 for Deep Learning Training paper arxiv.orgempirical study establishing bfloat16 for DL training
Evidence
5 spans quoted from the sources, strongest treatment first.
Released under Apache 2.0 with BF16, FP8, NVFP4, and GGUF weights on Hugging Face
BF16 precision for attention linear layers, MoE latent projection layers
select layers (including latent projections, MTP layers, QKV/attention projections, and embeddings) are maintained in BF16 or MXFP8 for training stability
we kept the QKV and attention projections in BF16
our tests show minimal intelligence impact compared to BF16 testing
Filed alongside
Other methods under optimization :: training precision.