Study Reveals Large Language Models Require Significantly More Training Data
Researchers find that current large language models are severely undertrained, proving that model size and training data must scale equally for optimal performance. The resulting Chinchilla model outperforms massive rivals like GPT-3 using a fraction of the computing power.
Researchers investigate the optimal model size and training token count for transformer language models under a fixed compute budget. They discover that current large language models suffer from significant undertraining, which stems from a recent industry trend of scaling up model sizes while keeping the amount of training data constant.
By training over 400 language models ranging from 70 million to over 16 billion parameters, the researchers determine that compute-optimal training requires scaling model size and the number of training tokens equally. This means that for every doubling of model size, developers must also double the number of training tokens.
The team tests this hypothesis by creating Chinchilla, a 70-billion-parameter model trained on four times more data than the 280-billion-parameter Gopher using the exact same compute budget. Chinchilla uniformly outperforms Gopher, GPT-3, Jurassic-1, and Megatron-Turing NLG across a wide range of evaluation tasks while requiring substantially less compute for fine-tuning and inference.