Model techniques map
Taxonomyinference & servinginference quantization

taxonomy node · level 2

inference quantization

64 methods filed at this node or below it, from the sources of 23 models.

inference & serving :: inference quantization

Matching aids for the classifier: FP8 quantization; NVFP4 quantization; MXFP4 weights; AWQ; GGUF quantization; W4A8; KV cache quantization; weight scaling; random Hadamard transform; QuaRot.

In this branch 64

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 64

Quantization core · 3 sources · 3 quotes
FP4 KV-cache quantization core · 2 sources · 5 quotes
Dynamic activation scaling core · 1 source · 1 quote
Mixed-FP8 layers in an NVFP4 recipe core · 1 source · 1 quote
Mobile-specialized quantization schema core · 1 source · 1 quote
NVFP4 quantization for routed-expert GEMMs core · 1 source · 1 quote
Offline weight-layout permutation core · 1 source · 1 quote
Per-tensor FP8 quantization core · 1 source · 1 quote
Selective retention of BF16 precision core · 1 source · 1 quote
FP8 default · 14 sources · 14 quotes
NVFP4 quantization default · 5 sources · 5 quotes
NVFP4 KV-cache quantization default · 1 source · 2 quotes
Post-RoPE KV-cache quantization default · 1 source · 1 quote
FP8 KV-cache quantization used · 13 sources · 15 quotes
MXFP4 weight quantization used · 3 sources · 3 quotes
Post-training quantization used · 3 sources · 3 quotes
Four-Over-Six used · 2 sources · 3 quotes
GGUF used · 2 sources · 2 quotes
MXFP8 activation quantization used · 2 sources · 2 quotes
SSM cache quantization used · 2 sources · 2 quotes
AWQ INT4 weight quantization (W4A16) used · 1 source · 1 quote
BF16 inference used · 1 source · 1 quote
Block-wise E4M3 FP8 weight quantization used · 1 source · 1 quote
Channel-wise quantization used · 1 source · 1 quote
Embedding and KV-cache quantization used · 1 source · 1 quote
Flex_AWQ_SSZ used · 1 source · 1 quote
FP16 cache storage with stochastic rounding used · 1 source · 1 quote
FP16 multimodal projector used · 1 source · 1 quote
FP4 precision for the attention indexer used · 1 source · 1 quote
FP8 inference used · 1 source · 1 quote
FP8 low-precision speculative draft computation used · 1 source · 1 quote
FP8 W8A8 quantization with dynamic activation scaling used · 1 source · 1 quote
Heuristic mixed per-layer precision quantization used · 1 source · 1 quote
Mixed-precision INT4/INT8 layer-wise quantization used · 1 source · 1 quote
ModelOpt FP4 quantization used · 1 source · 1 quote
MXFP4 tensor packing used · 1 source · 1 quote
MXFP8 block-scale regrouping at load used · 1 source · 1 quote
NVFP4 quantization with modelopt used · 1 source · 1 quote
NVIDIA ModelOpt quantization used · 1 source · 1 quote
Post-quantization of MoE model layers used · 1 source · 1 quote
QuaRot used · 1 source · 1 quote
Random Hadamard transform used · 1 source · 1 quote
SpinQuant R1 rotation used · 1 source · 1 quote
Static activation quantization used · 1 source · 1 quote
Targeted 2-bit quantization used · 1 source · 1 quote
TRT-LLM inference optimization to NVFP4 on Blackwell used · 1 source · 1 quote
W4A16 quantization used · 1 source · 1 quote
W4A8 quantization used · 1 source · 1 quote
NVFP4 ModelOpt re-quantization optional · 2 sources · 2 quotes
Fine-grained FP8 quantization optional · 1 source · 1 quote
IQ4_XS quantization optional · 1 source · 1 quote
Large-model compression using quantization and speculative sampling optional · 1 source · 1 quote
Mobile quantization optional · 1 source · 1 quote
Q3_K_L quantization optional · 1 source · 1 quote
Q4_0 quantization optional · 1 source · 1 quote
Q4_K_S quantization optional · 1 source · 1 quote
Block-scaled INT8 quantization with stochastic rounding evaluated · 2 sources · 2 quotes
FP8 E4M3 quantization evaluated · 2 sources · 2 quotes
Max-based scaling evaluated · 2 sources · 2 quotes
MSE-based scaling evaluated · 2 sources · 2 quotes
Empirical bits-per-element budget selection evaluated · 1 source · 1 quote
MSE calibration evaluated · 1 source · 1 quote
Low-bit quantization mentioned · 1 source · 1 quote
In-flight block-wise FP8 weight quantization not used · 1 source · 1 quote

By model

Which of this branch's techniques each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modeltechniques
GLM-5.3-Flash defaultFP8 defaultMXFP8 block-scale regrouping at load usedFP8 KV-cache quantization optional—
DeepSeek-V4.1-Flash coreFP4 KV-cache quantization corePost-RoPE KV-cache quantization default—
Hy4-preview usedFP8 usedLarge-model compression using quantization and speculative sampling optional—
NVIDIA-Nemotron-3-Ultra-550B-A55B coreMixed-FP8 layers in an NVFP4 recipe coreNVFP4 quantization for routed-expert GEMMs corePer-tensor FP8 quantization coreSelective retention of BF16 precision coreNVFP4 KV-cache quantization defaultFour-Over-Six usedFP16 cache storage with stochastic rounding usedFP8 usedFP8 KV-cache quantization usedHeuristic mixed per-layer precision quantization usedPost-training quantization usedRandom Hadamard transform usedSSM cache quantization usedW4A16 quantization usedBlock-scaled INT8 quantization with stochastic rounding evaluatedEmpirical bits-per-element budget selection evaluatedFP8 E4M3 quantization evaluatedMax-based scaling evaluatedMSE calibration evaluatedMSE-based scaling evaluated—
MiMo-V2.6-Flash coreDynamic activation scaling coreFP8 low-precision speculative draft computation used—
DeepSeek-V4-Flash usedFP4 precision for the attention indexer usedFP8 KV-cache quantization used—
MiMo-V2.5 usedBlock-wise E4M3 FP8 weight quantization usedFP8 used—
GLM-5.3 defaultFP8 defaultFP8 KV-cache quantization usedNVFP4 ModelOpt re-quantization optional—
Hy3 usedFP8 usedFP8 KV-cache quantization optionalQuantization optionalLow-bit quantization mentioned—
GLM-5.2 usedFlex_AWQ_SSZ usedFP8 inference usedQuaRot usedW4A8 quantization used—
DeepSeek-V4-Flash-Vision-Exp optionalFP8 KV-cache quantization optional—
DeepSeek-V4-Pro usedFP4 precision for the attention indexer usedFP8 KV-cache quantization used—
Gemma 4 31B coreMobile-specialized quantization schema coreQuantization coreChannel-wise quantization usedEmbedding and KV-cache quantization usedStatic activation quantization usedTargeted 2-bit quantization usedMobile quantization optionalQ4_0 quantization optionalPost-training quantization not used—
Inkling defaultNVFP4 quantization default—
Kimi K3 coreOffline weight-layout permutation coreMXFP4 weight quantization usedMXFP8 activation quantization used—
Laguna-S-2.1 usedAWQ INT4 weight quantization (W4A16) usedFP8 KV-cache quantization usedFP8 W8A8 quantization with dynamic activation scaling usedMixed-precision INT4/INT8 layer-wise quantization usedPost-quantization of MoE model layers usedSpinQuant R1 rotation usedTRT-LLM inference optimization to NVFP4 on Blackwell usedIn-flight block-wise FP8 weight quantization not used—
MiMo-V2.6-Pro coreDynamic activation scaling coreFP8 low-precision speculative draft computation used—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B usedPost-training quantization usedNVFP4 quantization optional—
Qwen3.5-397B-A17B usedFP8 usedNVFP4 quantization used—
Qwen3.6-35B-A3B usedFP8 KV-cache quantization usedFP8 optionalNVFP4 ModelOpt re-quantization optionalNVFP4 quantization optional—
Qwen3.8-Flash-Next optionalFine-grained FP8 quantization optional—
Step-3.7-Flash usedFP16 multimodal projector usedFP8 usedFP8 KV-cache quantization usedGGUF usedModelOpt FP4 quantization usedNVFP4 quantization usedNVFP4 quantization with modelopt usedNVIDIA ModelOpt quantization usedIQ4_XS quantization optionalQ3_K_L quantization optionalQ4_K_S quantization optional—
gpt-oss-120b usedBF16 inference usedMXFP4 tensor packing usedMXFP4 weight quantization used—