Open Models Almanac
Methodspost-trainingreward modelling

single mention · post-training

Environment-specific output-format reward signals

Environment-specific output-format reward signals grade whether a model's outputs meet each environment's formatting requirements, such as well-formed tool calls and consistent language.1 Each signal scales the positive environment reward multiplicatively, so format defects reduce reward while the environment reward remains the primary training signal.2

Trade-off The aggregation lets a single format defect significantly affect reward while ensuring environment reward stays primary even with multiple simultaneous defects.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we expect the model to produce well-formatted out- puts by, for example, making well-formed tool calls and keeping its language consistent across thinking and content blocks — tech report
  2. Format signals act multiplicatively on environment reward. They are only applied to positive rewards — tech report
  3. allows a single format signal to have a significant impact on the final reward, while ensuring that environment reward remains the primary training signal — tech report

Also called format signals.

Filed under post-training › reward modelling