Open Models Almanac
Methodstraining objectiveauxiliary loss

single mention · training objective

Attention-mask isolation of packed examples

Separates packed training examples so tokens from one example cannot condition on another.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See auxiliary loss for the methods filed beside it.

Also called training attention mask.

Filed under training objective › auxiliary loss