Open Models Almanac
Methodsother

single source · other

Re-render rollout context into training segments

Re-rendering splits a rollout sequence into segments whenever tokenization or chat-template changes alter the previously rendered context, instead of appending new tokens to the unchanged token-in/token-out context.1 Training then computes losses only on tokens the model newly generates in each segment, masking the re-rendered context and system, user and tool-response tokens.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. We split sequences into segments due to, e.g., chat-template changes that adapt previously rendered contexts — tech report
  2. We only train on the tokens that are newly generated by the model in each segment, masking out re-rendered context as well as system, user and tool-response tokens — tech report

Also called Segments.

Filed under other