Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attentionglobal attention

specific method · filed under model architecture

Global attention

Global attention is an attention mechanism in which each token can attend over the entire sequence rather than only a local window.1 It is interleaved with sliding window attention layers in hybrid architectures, motivating hybrids that combine local efficiency with full-sequence access.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. Sliding window attention [beltagy2020longformer] restricts each token’s attention scope to a local window rather than the entire sequence — MiMo-V2-Flash Technical Report
  2. This naturally motivates hybrid attention architectures that interleave sliding window attention with global attention — MiMo-V2-Flash Technical Report

Also called Global Attention (GA), Global attention layers.

sources
6
models
3
labs adopt it
2
strongest
core

Documented in

Evidence

6 quotes from 6 sources, one card per model, each opening with its strongest.

How sources treat it
used 3core 3

MiMo-V2.5

Xiaomiadopted3 sources

interleaving Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio and 128 sliding window

usedmodel architecturein MiMo-V2.5Xiaomi
2 more from this model

interleaving Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio

usedmodel architecturein MiMo-V2.5Xiaomi

global attention

coremodel architecturein MiMo-V2-FlashXiaomi
↑