Google Unveils PaLM: 540-Billion Parameter Language Model Powered by Pathways
Google introduces PaLM, a massive 540-billion parameter language model trained using its new Pathways system. The model achieves state-of-the-art few-shot performance across hundreds of tasks by efficiently scaling across thousands of TPU v4 chips.
Google Research introduces the Pathways Language Model (PaLM), a dense decoder-only Transformer model containing 540 billion parameters. This massive AI system trains efficiently using the new Pathways orchestration system, which allows a single model to generalize across multiple domains and tasks. By pushing the limits of model scale, Google aims to understand the new capabilities that emerge in few-shot learning scenarios.
PaLM represents the first large-scale implementation of the Pathways system, scaling training across 6,144 chips split between two Cloud TPU v4 Pods. This configuration marks the largest TPU-based system ever used for training. The system utilizes data parallelism at the Pod level alongside standard data and model parallelism within each individual Pod, achieving a significant increase in scale compared to previous large language models.
When evaluated on hundreds of language understanding and generation tasks, PaLM achieves state-of-the-art few-shot performance and outperforms previous models by significant margins. As the scale of the model increases, the performance improves consistently across all evaluated tasks while simultaneously unlocking entirely new emergent capabilities. This breakthrough demonstrates the immense potential of combining massive neural networks with highly efficient distributed computing infrastructure.