Open Models Almanac
Methodsother

single source · other

DARE

DARE employs random sparsification as an alternative to magnitude-based pruning.1 It drops parameters in a task vector with a specific probability and rescales the retained parameters to maintain the expected magnitude of the merged updates.2,3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. DARE (Drop And REscale) (Yu et al., 2024) employs random sparsification as an alternative to magnitude-based pruning — LFM2 Technical Report
  2. Each parameter in a task vector is dropped with probability (the drop rate) and retained otherwise — LFM2 Technical Report
  3. To maintain the expected magnitude of the merged updates, retained parameters are rescaled by — LFM2 Technical Report

Also called DARE model merging.

Filed under other