Microsoft and Nvidia Unveil 530-Billion Parameter MT-NLG AI Model
Microsoft and Nvidia collaborate to create MT-NLG, a massive 530-billion parameter language model that outperforms predecessors in natural language tasks. The system requires thousands of GPUs and advanced software tricks to train efficiently.
Microsoft and Nvidia announce the Megatron-Turing Natural Language Generation model, a massive AI system featuring 530 billion parameters. This collaboration combines Microsoft's Turing-NLG and Nvidia's Megatron-LM technologies to create a model three times larger than its predecessors, surpassing the scale of OpenAI's GPT-3.
The massive scale allows MT-NLG to excel at various natural language tasks without requiring extensive fine-tuning. It performs auto-completion, question answering, and reading comprehension through few-shot or zero-shot learning, proving that bigger neural networks generally deliver better and more adaptable results.
Training such a colossal model requires extreme computational power and clever software engineering. The companies use Nvidia's $85 million Selene supercomputer, which links 4,480 A100 GPUs to process 1.5 terabytes of data in just over a month, relying on the DeepSpeed library to parallelize the immense workload efficiently.