Open Models Almanac
Methodssoftware implementationtraining framework

specific method · filed under software implementation

Megatron-LM

Megatron-LM is a training and inference framework for large models that supports multi-node distributed parallelism and mixture-of-experts models.1 It shards a model across multiple GPUs and nodes, using expert, data, and context parallelism to speed up calibration and scale to long sequences.2,3

Trade-off Compared with single-process HuggingFace inference, it runs much faster on multi-node compute but requires that kind of distributed setup.4

How this definition was written

Summarised by glm-5.3-flash from 1 source; every sentence is checked against the quote it cites.

  1. Megatron-LM supports multi-node inference and n-D parallelism — NVIDIA Nemotron 3 Ultra Technical Report
  2. shard the model across multiple GPUs and across nodes — NVIDIA Nemotron 3 Ultra Technical Report
  3. expert parallelism and data parallelism speed up calibration by sharding experts across GPUs — NVIDIA Nemotron 3 Ultra Technical Report
  4. HuggingFace layerwise PTQ taking around 2 hours vs Megatron-LM taking 42 minutes on Ultra — NVIDIA Nemotron 3 Ultra Technical Report
sources
3
model
1
lab adopt it
1
strongest
used

Documented in

Evidence

3 quotes from 3 sources, one card per model, each opening with its strongest.

How sources treat it
used 3

NVIDIA-Nemotron-3-Ultra-550B-A55B

NVIDIAadopted3 sources

Megatron-LM allowed us to experiment faster and perform more complex long-sequence experiments on multi-node compute

usedsoftware implementationin Nemotron 3 UltraNVIDIA
2 more from this model

we chose Megatron-LM for its efficient multi-node distributed parallelism and MoE support

usedsoftware implementationin Nemotron 3 UltraNVIDIA

Software used for pre-training: Megatron-LM

usedsoftware implementationin Nemotron 3 UltraNVIDIA

Further reading

Picked by hand, not extracted: where to read more, not evidence for anything on this page.

↑