Open Models Almanac
Methodsmodel architecturetoken mixer

single mention · model architecture

Cross-layer attention sharing

Cross-layer attention sharing is a KV-cache sharing method in which some attention layers reuse the keys and values produced by earlier layers instead of maintaining their own.1 It acts as a KV-cache compression mechanism that reduces what must be stored, with affected layers taking a similar residual stream input to the producing layer.2,3

Trade-off Removing CLA sacrifices inference optimizations in exchange for model performance closer to a full-attention model.4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. it is more difficult to remove CLA and YOCO KV-cache sharing: layers that previously relied on keys and values produced by earlier layers then need to compute and maintain their own KV caches — technical report
  2. The goal was to remove KV-cache compression mechanisms — technical report
  3. which is close to the affected layer and takes a very similar residual stream as input — technical report
  4. The goal was to remove KV-cache compression mechanisms to achieve an architecture as close as possible to a full-attention model. Although this approach temporarily sacrificed inference optimizations, we prioritized model performance — technical report

Also called Cross-layer attention sharing (CLA), attention sharing method.

Filed under model architecture › token mixer