Open Models Almanac
Methodsoptimizationtraining parallelism

single mention · optimization

KDA Context Parallelism

KDA Context Parallelism splits a sequence across context-parallel ranks when training or prefilling Kimba delta-rule linear attention, where prior direct-summation methods fail.1,2 Each segment's effect is decomposed into two locally computable quantities, a cumulative transition acting on the incoming state and a state generated from zero, which ranks exchange to reconstruct exact states.3,4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. we introduce KDA Context Parallelism (KCP), which decomposes the effect of each — k3 tech report
  2. This direct summation, however, is insufficient for KDA — k3 tech report
  3. decomposes the effect of each segment into two locally computable quantities, a cumulative transition acting on the incoming state and a state generated locally from zero — k3 tech report
  4. are the fragments each rank exchanges with the others — k3 tech report

Also called KDA Context Parallelism (KCP).

Filed under optimization › training parallelism