single mention · optimization
KDA Context Parallelism
KDA Context Parallelism splits a sequence across context-parallel ranks when training or prefilling Kimba delta-rule linear attention, where prior direct-summation methods fail.1,2 Each segment's effect is decomposed into two locally computable quantities, a cumulative transition acting on the incoming state and a state generated from zero, which ranks exchange to reconstruct exact states.3,4
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
we introduce KDA Context Parallelism (KCP), which decomposes the effect of each
— k3 tech reportThis direct summation, however, is insufficient for KDA
— k3 tech reportdecomposes the effect of each segment into two locally computable quantities, a cumulative transition acting on the incoming state and a state generated locally from zero
— k3 tech reportare the fragments each rank exchanges with the others
— k3 tech report
Also called KDA Context Parallelism (KCP).
Filed under optimization › training parallelism