Open Models Almanac
Methodsmodel architecturetoken mixerhybrid layer stacking

single mention · model architecture

Dense Attention Fallback

Uses dense attention as a per-layer alternative when the usual attention pattern is not applied.

The sources on this page only mention it in passing. The line above is the extraction's one-line gloss. See hybrid layer stacking for the methods filed beside it.

Also called dense attention as a per-layer fallback.

Filed under model architecture › token mixer › hybrid layer stacking