Open Models Almanac
Methodsmodel architecturetoken mixersoftmax attention

general family · filed under model architecture

Bidirectional attention

Bidirectional attention lets each action token attend to all other tokens in its block, rather than only to current and previous tokens as with a causal mask.1,2 It is used for parallel decoding of action chunks, so the model can leverage all intermediate features non-causally when predicting each element of the chunk.3,4

Trade-off It enables parallel decoding, but in one comparison bidirectional interaction between action tokens harmed performance relative to causal self-attention.5

How this definition was written

Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.

  1. A causal attention mask ensures the model only attends to current and previous tokens — Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
  2. We also use a bidirectional attention mask (instead of causal — Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
  3. replaces causal attention with bidirectional attention for parallel decoding — Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
  4. enabling the model to leverage all intermediate features non-causally when predicting each element in the action chunk — Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
  5. Table 7 shows causal self-attention performs best, while bidirectional interaction harms performance — SmolVLA: A vision-language-action model for affordable and efficient robotics

Also called full bidirectional attention, bidirectional self-attention.

sources
4
models
3
labs adopt it
2
strongest
core

Documented in

OpenVLA project (Stanford et al.)
Physical Intelligence
Hugging Face

Evidence

4 quotes from 4 sources, one card per model, each opening with its strongest.

How sources treat it
evaluated 1used 1core 2

π0.5

Physical Intelligenceadopted2 sources

image patch, textual prompt, and continuous action tokens use bidirectional attention

usedunclearin π0.5
1 more from this model

Within each block, there is full bidirectional attention

coremodel architecturein π0Physical Intelligence
↑