single source · other
DELLA
DELLA is a method that extends DARE by replacing uniform random dropout with magnitude-aware sampling.1 It ranks parameters by absolute magnitude and assigns dropout probabilities inversely proportional to rank, as lower-magnitude parameters can tolerate higher dropout rates.2,3
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
DELLA (Drop and rEscaLe via sampLing with mAgnitude) (Deep et al., 2024) extends DARE by replacing uniform random dropout with magnitude-aware sampling
— LFM2 Technical ReportFor each layer, parameters are ranked by absolute magnitude, and dropout probabilities are assigned inversely proportional to rank
— LFM2 Technical ReportThe key insight is that lower-magnitude parameters contribute less to task-specific performance and can tolerate higher dropout rates
— LFM2 Technical Report
Also called DELLA model merging.
Filed under other