single mention · not yet filed
The sources on this page only mention it in passing.
Methods the sources say are used for this one.
GDN compresses the prefix into a fixed-size recurrent state and updates that state according to the current content, while the interleaved global-attention layers
We place one such full-attention layer in every four layers, with the other three layers using GDN.