π0.5
Physical Intelligenceimage patch, textual prompt, and continuous action tokens use bidirectional attention
1 more from this model
Within each block, there is full bidirectional attention
general family · filed under model architecture
Bidirectional attention lets each action token attend to all other tokens in its block, rather than only to current and previous tokens as with a causal mask.1,2 It is used for parallel decoding of action chunks, so the model can leverage all intermediate features non-causally when predicting each element of the chunk.3,4
Trade-off It enables parallel decoding, but in one comparison bidirectional interaction between action tokens harmed performance relative to causal self-attention.5
Summarised by glm-5.3-flash from 2 sources; every sentence is checked against the quote it cites.
A causal attention mask ensures the model only attends to current and previous tokens— Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
We also use a bidirectional attention mask (instead of causal— Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
replaces causal attention with bidirectional attention for parallel decoding— Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
enabling the model to leverage all intermediate features non-causally when predicting each element in the action chunk— Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
Table 7 shows causal self-attention performs best, while bidirectional interaction harms performance— SmolVLA: A vision-language-action model for affordable and efficient robotics
Also called full bidirectional attention, bidirectional self-attention.
4 quotes from 4 sources, one card per model, each opening with its strongest.
image patch, textual prompt, and continuous action tokens use bidirectional attention
Within each block, there is full bidirectional attention
We also use a bidirectional attention mask (instead of causal), enabling the model to leverage all intermediate features non-causally when predicting each element in the action chunk.
bidirectional interaction harms performance.