Open Models Almanac
Methodsinference & servinginference quantization

methods · inference & serving

inference quantization

112 methods filed at this node or below it, from the sources of 27 models.

In this branch 112

Everything filed at this node or below it, with one collapsible heading per child node.

filed here 112

Quantization core · 4 sources · 4 quotes
FP4 global KV-cache quantization core · 2 sources · 2 quotes
Mobile quantization core · 2 sources · 2 quotes
Per-tensor FP8 quantization core · 2 sources · 2 quotes
FP8 default · 18 sources · 18 quotes
BF16 inference default · 2 sources · 2 quotes
Mixed-precision NVFP4 routed-expert quantization default · 2 sources · 2 quotes
FP8 KV-cache quantization used · 12 sources · 13 quotes
Post-training quantization used · 5 sources · 5 quotes
NVFP4 quantization used · 4 sources · 4 quotes
MSE-based weight-scale selection used · 3 sources · 4 quotes
MXFP4 weight quantization used · 3 sources · 3 quotes
Four-Over-Six used · 2 sources · 3 quotes
GGUF used · 2 sources · 2 quotes
Mamba SSM-cache quantization used · 2 sources · 2 quotes
ModelOpt NVFP4 re-quantization used · 2 sources · 2 quotes
MXFP8 activation quantization used · 2 sources · 2 quotes
FP8 E4M3 quantization evaluated · 2 sources · 2 quotes
Max-based weight scaling evaluated · 2 sources · 2 quotes
92 from one source, or only mentioned as not used
Backbone-coupled MTP KV-cache precision core · 1 source · 1 quote
Dynamic activation scaling core · 1 source · 1 quote
Mixed-FP8 layers in an NVFP4 recipe core · 1 source · 1 quote
Mixed-precision inference core · 1 source · 1 quote
Native 2-bit quantization core · 1 source · 1 quote
NVFP4 routed-expert GEMM quantization core · 1 source · 1 quote
Offline weight-layout permutation core · 1 source · 1 quote
FP8 latent KV-cache quantization default · 1 source · 2 quotes
NVFP4 KV-cache quantization default · 1 source · 2 quotes
Abs-max quantization used · 1 source · 1 quote
AWQ INT4 weight-only quantization used · 1 source · 1 quote
Channel-wise quantization used · 1 source · 1 quote
Component-wise precision selection used · 1 source · 1 quote
Embedding and KV-cache quantization used · 1 source · 1 quote
Flex_AWQ_SSZ used · 1 source · 1 quote
FP16 multimodal projector used · 1 source · 1 quote
FP4 attention-indexer precision used · 1 source · 1 quote
FP8 precision used · 1 source · 1 quote
FP8-or-higher lm_head precision used · 1 source · 1 quote
Layer-wise mixed INT4/INT8 quantization used · 1 source · 1 quote
ModelOpt FP4 quantization used · 1 source · 1 quote
MoE-layer quantization used · 1 source · 1 quote
MXFP4 block-scaled tensor packing used · 1 source · 1 quote
MXFP8 block-scale regrouping at load used · 1 source · 1 quote
Native FP8 KV cache with offloading used · 1 source · 1 quote
NVFP4 post-training quantization used · 1 source · 1 quote
NVFP4 W4A4 quantization of MTP experts used · 1 source · 1 quote
NVIDIA ModelOpt quantization used · 1 source · 1 quote
OCP MXFP4 quantization used · 1 source · 1 quote
Parameter alignment for quantization used · 1 source · 1 quote
Platform-dependent W4A4/W4A16 quantization used · 1 source · 1 quote
Post-training calibration sample count used · 1 source · 1 quote
Quantization-aware distillation used · 1 source · 1 quote
QuaRot used · 1 source · 1 quote
Random Hadamard transform used · 1 source · 1 quote
SFT-mixture calibration used · 1 source · 1 quote
SpinQuant R1 rotation preprocessing used · 1 source · 1 quote
Static activation quantization used · 1 source · 1 quote
Targeted 2-bit quantization used · 1 source · 1 quote
W4A16 quantization used · 1 source · 1 quote
W4A8 quantization used · 1 source · 1 quote
Fine-grained FP8 quantization optional · 1 source · 1 quote
IQ4_XS optional · 1 source · 1 quote
NVFP4 quantization of MoE expert linears optional · 1 source · 1 quote
NVFP4 weight-and-activation quantization optional · 1 source · 1 quote
Q3_K_L optional · 1 source · 1 quote
Q4_0 optional · 1 source · 1 quote
Q4_K_S optional · 1 source · 1 quote
Separate target and MTP lm_head precision optional · 1 source · 1 quote
4-bit quantization evaluated · 1 source · 1 quote
8-bit and 4-bit quantization evaluated · 1 source · 1 quote
8-bit quantization evaluated · 1 source · 1 quote
BF16 expert activations evaluated · 1 source · 1 quote
Block-scaled NVFP4 KV-cache quantization evaluated · 1 source · 1 quote
Empirical bits-per-element budget selection evaluated · 1 source · 1 quote
FP8 or NVFP4 KV-cache quantization evaluated · 1 source · 1 quote
Linear E2M1 residual correction evaluated · 1 source · 1 quote
Promote expert weights from NVFP4 to FP8 evaluated · 1 source · 2 quotes
Quantized and reduced-precision inference evaluated · 1 source · 1 quote
Restricted-search MSE weight-scale selection evaluated · 1 source · 1 quote
Sensitivity-driven mixed-precision quantization evaluated · 1 source · 1 quote
Low-bit quantization mentioned · 1 source · 1 quote
Rotation-based quantization mitigation mentioned · 1 source · 1 quote
Mixed-precision key/value cache quantization not used · 1 source · 1 quote
On-the-fly block-wise FP8 weight quantization not used · 1 source · 1 quote
W4A4 quantization not used · 1 source · 1 quote

By model

Which of this branch's methods each model's own documents describe, and how strongly. Under each model: its strongest treatment anywhere in the branch.

Modelmethods
GLM-5.3-Flash defaultFP8 defaultMXFP8 block-scale regrouping at load usedFP8 KV-cache quantization optional—
DeepSeek-V4.1-Flash coreFP4 global KV-cache quantization core—
Hy4-preview usedFP8 usedLarge-model quantization and speculative sampling optional—
Step-5-Preview optionalFP8 optional—
GLM-5.3 defaultFP8 defaultFP8 KV-cache quantization usedNVFP4 quantization of MoE expert linears optional—
Hy3 usedFP8 usedFP8 KV-cache quantization optionalQuantization optionalLow-bit quantization mentioned—
MiMo-V2.6-Flash coreDynamic activation scaling coreFP8 low-precision speculative draft computation used—
NVIDIA-Nemotron-3-Ultra-550B-A55B coreMixed-FP8 layers in an NVFP4 recipe coreNVFP4 routed-expert GEMM quantization corePer-tensor FP8 quantization coreSelective BF16 retention for sensitive layers coreNVFP4 KV-cache quantization defaultFour-Over-Six usedFP16 cache storage with stochastic rounding usedFP8 usedFP8 KV-cache quantization usedHeuristic mixed per-layer precision quantization usedMamba SSM-cache quantization usedPost-training quantization usedRandom Hadamard transform usedW4A16 quantization usedBlock-scaled INT8 quantization with stochastic rounding evaluatedEmpirical bits-per-element budget selection evaluatedFP8 E4M3 quantization evaluatedMax-based weight scaling evaluatedMSE-based weight-scale selection evaluated—
DeepSeek-V4-Flash usedFP4 attention-indexer precision usedFP8 precision used—
DeepSeek-V4-Flash-Vision-Exp optionalFP8 KV-cache quantization optional—
DeepSeek-V4-Pro usedFP4 attention-indexer precision usedFP8 precision used—
GLM-5.2 usedFlex_AWQ_SSZ usedFP8 usedQuaRot usedW4A8 quantization used—
Gemma 4 31B coreMobile quantization coreQuantization coreChannel-wise quantization usedEmbedding and KV-cache quantization usedStatic activation quantization usedTargeted 2-bit quantization usedQ4_0 optionalPost-training quantization not used—
Inkling defaultMixed-precision NVFP4 routed-expert quantization default—
Kimi K3 coreOffline weight-layout permutation coreMXFP4 weight quantization usedMXFP8 activation quantization used—
Laguna-S-2.1 usedAWQ INT4 weight-only quantization usedFP8 KV-cache quantization usedFP8 KV-cache quantization with amax-based scaling usedFP8 W8A8 quantization with dynamic activation scaling usedLayer-wise mixed INT4/INT8 quantization usedMoE-layer quantization usedSpinQuant R1 rotation preprocessing usedTRT-LLM inference optimization to NVFP4 on Blackwell usedOn-the-fly block-wise FP8 weight quantization not used—
MiMo-V2.5 usedFP8 used—
MiMo-V2.6-Pro coreDynamic activation scaling coreFP8 low-precision speculative draft computation used—
Mistral-Large-4 defaultFP8 latent KV-cache quantization defaultAbs-max quantization usedFP8 usedMixed-precision NVFP4 routed-expert quantization usedNative FP8 KV cache with offloading usedOCP MXFP4 quantization usedSFT-mixture calibration usedLayer-by-layer FP8-to-MXFP4 expert re-quantization at load optional—
NVIDIA-Nemotron-3.5-Lightning-30B-A3B coreBackbone-coupled MTP KV-cache precision coreShared FP8 lm_head for target verification and MTP drafting defaultAggressive mixed-precision quantization recipe usedBF16 MTP attention weights with FP8 K/V BMM quantizers usedFP8-or-higher lm_head precision usedGradient-based per-layer mixed-precision quantization search usedMSE-based weight-scale selection usedNVFP4 post-training quantization usedNVFP4 W4A4 quantization of MTP experts usedNVFP4 weight quantization with BF16 activations usedParameter alignment for quantization usedPer-block FP8 quantization for attention Q/K/V projections usedPost-training calibration sample count usedPost-training quantization usedQuantization-aware distillation usedSeparate MTP quantization and backbone splicing usedTask-loss-gradient-weighted quantization sensitivity estimation usedWeight-only post-training quantization for parallel drafters usedNVFP4 quantization optionalSeparate target and MTP lm_head precision optionalBF16 expert activations evaluatedBlock-scaled NVFP4 KV-cache quantization evaluatedFP4 quantization of Mamba output-projection weights evaluatedFP8 or NVFP4 KV-cache quantization evaluatedIncrease MoE expert precision and fine-tune on RL rollouts evaluatedLinear E2M1 residual correction evaluatedMXFP8 block-scaled attention projection quantization evaluatedPer-tensor FP8 quantization evaluatedPromote expert weights from NVFP4 to FP8 evaluatedQuantization of the parallel-drafter language-model head evaluatedRestricted-search MSE weight-scale selection evaluatedRotation or higher precision for Mamba input activations evaluatedSensitivity-driven mixed-precision quantization evaluatedTwo-pass FP4 weight quantization with residual correction evaluatedW4A16 quantization of Mamba output-projection weights evaluatedHigher-precision activations for Mamba output projection mentionedRotation-based quantization mitigation mentionedMixed-precision key/value cache quantization not usedW4A4 quantization not used—
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B coreBackbone-coupled MTP KV-cache precision coreMixed-precision inference coreNative 2-bit quantization coreShared FP8 lm_head for target verification and MTP drafting defaultAggressive mixed-precision quantization recipe usedBF16 MTP attention weights with FP8 K/V BMM quantizers usedComponent-wise precision selection usedFP8-or-higher lm_head precision usedGradient-based per-layer mixed-precision quantization search usedMSE-based weight-scale selection usedNVFP4 post-training quantization usedNVFP4 W4A4 quantization of MTP experts usedNVFP4 weight quantization with BF16 activations usedParameter alignment for quantization usedPer-block FP8 quantization for attention Q/K/V projections usedPlatform-dependent W4A4/W4A16 quantization usedPost-training calibration sample count usedPost-training quantization usedQuantization-aware distillation usedSeparate MTP quantization and backbone splicing usedTask-loss-gradient-weighted quantization sensitivity estimation usedWeight-only post-training quantization for parallel drafters usedNVFP4 weight-and-activation quantization optionalSeparate target and MTP lm_head precision optionalBF16 expert activations evaluatedBlock-scaled NVFP4 KV-cache quantization evaluatedFP4 quantization of Mamba output-projection weights evaluatedFP8 or NVFP4 KV-cache quantization evaluatedIncrease MoE expert precision and fine-tune on RL rollouts evaluatedLinear E2M1 residual correction evaluatedMXFP8 block-scaled attention projection quantization evaluatedPer-tensor FP8 quantization evaluatedPromote expert weights from NVFP4 to FP8 evaluatedQuantization of the parallel-drafter language-model head evaluatedRestricted-search MSE weight-scale selection evaluatedRotation or higher precision for Mamba input activations evaluatedSensitivity-driven mixed-precision quantization evaluatedTwo-pass FP4 weight quantization with residual correction evaluatedW4A16 quantization of Mamba output-projection weights evaluatedHigher-precision activations for Mamba output projection mentionedRotation-based quantization mitigation mentionedMixed-precision key/value cache quantization not usedW4A4 quantization not used—
OpenVLA / OpenVLA-OFT defaultBF16 inference defaultQuantization optional4-bit quantization evaluated8-bit and 4-bit quantization evaluated8-bit quantization evaluatedQuantized and reduced-precision inference evaluated—
Qwen3.5-397B-A17B usedFP8 usedNVFP4 quantization used—
Qwen3.6-35B-A3B usedFP8 KV-cache quantization usedFP8 optionalModelOpt NVFP4 re-quantization optionalNVFP4 quantization optional—
Qwen3.8-Flash-Next optionalFine-grained FP8 quantization optional—
Step-3.7-Flash usedFP16 multimodal projector usedFP8 usedFP8 KV-cache quantization usedGGUF usedModelOpt FP4 quantization usedModelOpt NVFP4 re-quantization usedNVFP4 quantization usedNVIDIA ModelOpt quantization usedIQ4_XS optionalQ3_K_L optionalQ4_K_S optional—
gpt-oss-120b usedBF16 inference usedMXFP4 block-scaled tensor packing usedMXFP4 weight quantization used—
↑