Open Models Almanac
Methodstraining objectivedistillation objective

single mention · training objective

Cross-encoder-to-ColBERT knowledge distillation

Cross-encoder-to-ColBERT knowledge distillation trains a ColBERT student model to match relevance scores computed by a cross-encoder teacher.1 It minimizes the mean squared error between student and teacher relevance scores, giving richer continuous training signals than binary contrastive labels.2

Trade-off Distillation yields better ColBERT models than pure contrastive learning by providing richer training signals through continuous relevance scores.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. leveraging pre-computed relevance scores from a powerful cross-encoder teacher model — LFM2 Technical Report
  2. minimizes the mean squared error between student and teacher relevance scores — LFM2 Technical Report
  3. richer training signals through continuous relevance scores rather than binary positive/negative labels — LFM2 Technical Report

Also called Knowledge distillation framework.

Filed under training objective › distillation objective