specific method · filed under software implementation
Megatron-LM
Megatron-LM is a training and inference framework for large models that supports multi-node distributed parallelism and mixture-of-experts models.1 It shards a model across multiple GPUs and nodes, using expert, data, and context parallelism to speed up calibration and scale to long sequences.2,3
Trade-off Compared with single-process HuggingFace inference, it runs much faster on multi-node compute but requires that kind of distributed setup.4
How this definition was written
Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.
Megatron-LM supports multi-node inference and n-D parallelism
— NVIDIA Nemotron 3 Ultra Technical Reportshard the model across multiple GPUs and across nodes
— NVIDIA Nemotron 3 Ultra Technical Reportexpert parallelism and data parallelism speed up calibration by sharding experts across GPUs
— NVIDIA Nemotron 3 Ultra Technical ReportHuggingFace layerwise PTQ taking around 2 hours vs Megatron-LM taking 42 minutes on Ultra
— NVIDIA Nemotron 3 Ultra Technical Report