Open Models Almanac
Methodsmodel architecturetoken mixerhybrid layer stacking

specific method · filed under model architecture

Interleaved Self-Attention and Cross-Attention

Alternates self-attention and cross-attention layers or blocks, rather than combining local and global self-attention.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See hybrid layer stacking for the methods filed beside it.

Also called interleaved design, interleaved SA+CA setup.

sources
2
model
1
lab adopt it
1
strongest
core

Documented in

Hugging Face

Evidence

4 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
core 4

SmolVLA

Hugging Faceadopted2 sources

we employ an interleaved approach, where each block contains either a CA or a SA layer.

coremodel architecturein SmolVLAHugging Face
3 more from this model

Inside the action expert, attention layers alternate between:

coremodel architecturein SmolVLAHugging Face

Interleaving self-attention and cross-attention blocks

coremodel architecturein SmolVLAHugging Face

our proposed interleaved SA+CA setup

coreunclearin SmolVLA
↑