Open Models Almanac
Methodsmodel architecturetoken mixersparse attentionsparse attention indexer

specific method · filed under model architecture

Fine-grained token selection mechanism

A fine-grained token selection mechanism is a sparse-attention component that, for each query token, selects which preceding tokens' key-value entries participate in attention.1 It retrieves only the key-value entries with the top-k index scores, computed by a lightning indexer, and applies attention between the query and those sparsely selected entries.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. determining which tokens to be selected by the query token — paper
  2. retrieves only the key-value entries {c𝑠} corresponding to the top-k index scores — paper
sources
2
model
1
lab adopt it
1
strongest
core

Documented in

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
core 2

DeepSeek-V3.2

DeepSeekadopted2 sources

our fine-grained token selection mechanism retrieves only the key-value entries {c_s} corresponding to the top-k index scores.

coremodel architecturein DeepSeek-V3.2DeepSeek
1 more from this model

our fine-grained token selection mechanism retrieves only the key-value entries corresponding to the top-k index scores.

coremodel architecturein DeepSeek-V3.2DeepSeek
↑