specific method · filed under software implementation
Megatron-LM
A framework used for distributed model pre-training and experiments across multiple nodes.
- sources
- 3
- model
- 1
- lab adopt it
- 1
- strongest
- used
How sources treat it
One count per evidence span, weakest treatment to strongest.
used 3
Documented in
Further reading
Picked by hand, not extracted: where to read more, not evidence for anything on this page.
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (Shoeybi et al., 2019) paper arxiv.orgintroduces Megatron-LM's tensor-parallel training
- NVIDIA/Megatron-LM code github.comofficial training framework repository
Evidence
3 spans quoted from the sources, strongest treatment first.
we chose Megatron-LM for its efficient multi-node distributed parallelism and MoE support
usedsoftware implementationin Nemotron 3 UltraNVIDIA
Software used for pre-training: Megatron-LM
usedsoftware implementationin Nemotron 3 UltraNVIDIA
Megatron-LM allowed us to experiment faster and perform more complex long-sequence experiments on multi-node compute
usedsoftware implementationin Nemotron 3 UltraNVIDIA
Filed alongside
Other methods under software implementation :: training framework.