Microsoft and NVIDIA Unveil 530-Billion Parameter Language Model
Microsoft and NVIDIA collaborate to build MT-NLG, a 530-billion parameter generative language model that sets new accuracy standards. The massive transformer model leverages DeepSpeed and Megatron to achieve unmatched performance across various natural language tasks.
Microsoft and NVIDIA introduce MT-NLG, a 530-billion parameter monolithic transformer language model that stands as the largest and most powerful of its kind trained to date. This model triples the parameter count of previous leading models and results from a deep research collaboration focused on parallelizing and optimizing the training of massive AI systems.
The 105-layer MT-NLG model achieves unmatched accuracy across a broad set of natural language tasks, including completion prediction, reading comprehension, commonsense reasoning, and word sense disambiguation. It sets a new standard for large-scale language models by outperforming prior state-of-the-art models in zero-, one-, and few-shot learning settings.
Transformer-based language models grow exponentially because larger models acquire a richer understanding of language and generalize better as zero- or few-shot learners. This enhanced capability unlocks exciting downstream applications such as automatic dialogue generation, summarization, translation, semantic search, and code autocompletion, even though training models of this massive scale presents significant computational challenges.