Microsoft and Nvidia Unveil 530-Billion Parameter MT-NLG AI Model
The tech giants collaborate to build MT-NLG, a massive language model that outperforms predecessors in various natural language tasks without needing extensive fine-tuning.
Nvidia and Microsoft introduce the Megatron-Turing Natural Language Generation model, a massive AI system featuring 530 billion parameters. This new model surpasses previous collaborative efforts like Microsoft's Turing-NLG and Nvidia's Megatron-LM by incorporating three times as many parameters across 105 layers. Compared to other industry models, MT-NLG sits between OpenAI's 175-billion-parameter GPT-3 and Google's 1.6-trillion-parameter Switch Transformer.
The increased size allows MT-NLG to excel at a wide variety of natural language tasks, including auto-completing sentences, answering questions, and reading comprehension. It performs these complex functions with little to no fine-tuning, a capability known as few-shot or zero-shot learning. In the realm of neural networks, this larger scale translates directly to improved reasoning and broader language understanding.
Training such a colossal model requires immense computational power, achieved using Nvidia's Selene supercomputer equipped with 4,480 A100 GPUs. The engineering teams process 1.5 terabytes of data over the course of a month by utilizing the DeepSpeed library to efficiently distribute workloads across the hardware. This careful coordination of tensor-slicing and pipeline parallelism ensures the massive system trains effectively without breaking down.