DeepSeek-V4-Flash-Vision-Exp
DeepSeekThe reference inference covers the vision encoder and aligner, DFlash attention, MoE, Hyper-Connections, and the DSpark forward path.
specific method · filed under model architecture
Hyper-Connections (HC) replaces the single residual vector with several parallel branches that widen the residual stream.1 It uses learnable read, write, and mixing operators, predicted from the residual state, to form block inputs, distribute block outputs over branches, and exchange information between branches.2,3
Trade-off HC adds parameters per block with negligible compute cost, the main extra cost being memory traffic from carrying multiple branches.4
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
replace the single residual vector with several parallel branches— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
A read operator forms the block input, a write operator distributes the block output over branches, and a mixing operator exchanges information between branches— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
The three operators are predicted from the residual state— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
This adds parameters per block and no matrix multiplication, so its compute cost is negligible; the extra cost is the memory traffic of carrying branches instead of one— On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
Also called Hyper-Connections (HC).
2 quotes from 2 sources, one card per model, each opening with its strongest.
The reference inference covers the vision encoder and aligner, DFlash attention, MoE, Hyper-Connections, and the DSpark forward path.
HC generalizes Eq. (21) and Eq. (22) into three learnable operators.