A natural alternative to the sampled-token objective is distribution-level distillation, where the student is trained to match the teacher’s predictive distribution
2 more from this model
In our preliminary experiments, these objectives did not improve MOPD performance and consistently underperformed the sampled-token objective
In our preliminary experiments, these objectives did not improve MOPD performance