Open Models Almanac
Methodsother

single source · other

DELLA

DELLA is a method that extends DARE by replacing uniform random dropout with magnitude-aware sampling.1 It ranks parameters by absolute magnitude and assigns dropout probabilities inversely proportional to rank, as lower-magnitude parameters can tolerate higher dropout rates.2,3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. DELLA (Drop and rEscaLe via sampLing with mAgnitude) (Deep et al., 2024) extends DARE by replacing uniform random dropout with magnitude-aware sampling — LFM2 Technical Report
  2. For each layer, parameters are ranked by absolute magnitude, and dropout probabilities are assigned inversely proportional to rank — LFM2 Technical Report
  3. The key insight is that lower-magnitude parameters contribute less to task-specific performance and can tolerate higher dropout rates — LFM2 Technical Report

Also called DELLA model merging.

Filed under other