Open Models Almanac
Methodsoptimizationtraining stability

implementation detail · filed under optimization

Weight padding

Weight padding pads affected weight matrices at load time so kernels see a shape that meets alignment requirements.1 It is used when tensor parallelism, quantization format, and hardware combine to give per-rank GEMM dimensions that violate kernel alignment requirements.2

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. the route we have taken is to pad the affected weight matrices at load time so the kernel sees an acceptable shape — Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
  2. Several combinations of tensor parallelism factor, quantization format, and target hardware land the per-rank GEMM dimensions at sizes that violate kernel alignment requirements — Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
sources
2
model
1
lab adopt it
1
strongest
used

Documented in

Evidence

2 quotes from 2 sources, one card per model, each opening with its strongest.

How sources treat it
used 2
↑