Open Models Almanac
Methodspost-training

single mention · post-training

Specialize-then-rehearse training

Specialize-then-rehearse training is a two-stage fine-tuning recipe that adapts an existing released checkpoint to a new domain instead of re-running full training from scratch.1 Stage 1 fine-tunes heavily on the new-domain corpus (plus some text data to preserve language ability), and Stage 2 continues training on a mixture that interleaves the new data with the original mid-training data to balance the two.2,3

Trade-off It targets a Pareto trade-off between new-domain benchmarks and the base model's general benchmarks, with the mixture ratio swept to find the best balance.4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. To avoid the redundant compute of re-running Molmo2’s full multimodal training with our new corpus integrated, we build on the released Molmo2 checkpoint with a two-stage specialize-then-rehearse recipe — MolmoAct2 Action Reasoning Models for Real-World Deployment
  2. we fine-tune for K steps on the Molmo2-ER corpus augmented with Tulu-3 (Lambert et al., 2025) text-only data to preserve language competence — MolmoAct2 Action Reasoning Models for Real-World Deployment
  3. a mixture that interleaves our embodied corpus with Molmo2’s original multimodal mid-training data — MolmoAct2 Action Reasoning Models for Real-World Deployment
  4. Sweeping , we find yields the best Pareto trade-off between the embodied-reasoning benchmarks in Table 3 and Molmo2’s general benchmarks — MolmoAct2 Action Reasoning Models for Real-World Deployment

Also called specialize-then-rehearse recipe.

Filed under post-training