Open Models Almanac
Methodsmodel architecturetoken mixersparse attentionsparse attention indexer

specific method · filed under model architecture

Hierarchical Sparse Indexer

A Hierarchical Sparse Indexer is a decoder sparse-attention component in which a first layer's index scores restrict the candidate positions that later indexers search.1 The first layer scores the full visible context and selects blocks into a candidate pool, so deeper indexers score only candidates rather than the whole context, reducing repeated scoring during decode.2

Trade-off For a fixed candidate-pool size, deeper indexers' per-query cost drops from linear in context length to constant.3

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. the first layer assigned to Full Mode constructs a candidate pool that later re-indexing layers use as their search domain — DeepSeek V41 Tech Report
  2. This first Full Mode layer scores all causally visible main KV positions and produces the Top-K indices for its own attention. It also performs blockwise candidate selection — DeepSeek V41 Tech Report
  3. this changes the per-query cost of deeper indexers from linear in context length to — DeepSeek V41 Tech Report

Also called Hierarchical sparse indexing.

sources
2
model
1
lab adopt it
1
strongest
core

Documented in

Evidence

3 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 1core 2

DeepSeek-V4.1-Flash

DeepSeekadopted2 sources

the first layer assigned to Full Mode constructs a candidate pool that later re-indexing layers use as their search domain

coreunclearin DeepSeek-V4.1-FlashDeepSeek
2 more from this model

a Hierarchical Sparse Indexer restricts later indexers to a candidate pool selected by an earlier indexer

coreunclearin DeepSeek-V4.1-FlashDeepSeek

In the decoder, a Hierarchical Sparse Indexer further restricts later indexing layers to a candidate pool constructed by the first Full Mode layer, bounding deeper indexer cost independently of context length.

usedmodel architecturein DeepSeek-V4.1-FlashDeepSeek
↑