single source · other
DARE
DARE employs random sparsification as an alternative to magnitude-based pruning.1 It drops parameters in a task vector with a specific probability and rescales the retained parameters to maintain the expected magnitude of the merged updates.2,3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
DARE (Drop And REscale) (Yu et al., 2024) employs random sparsification as an alternative to magnitude-based pruning
— LFM2 Technical ReportEach parameter in a task vector is dropped with probability (the drop rate) and retained otherwise
— LFM2 Technical ReportTo maintain the expected magnitude of the merged updates, retained parameters are rescaled by
— LFM2 Technical Report
Also called DARE model merging.
Filed under other