Google Unveils PaLM, a 540-Billion Parameter Language Model
Google Research introduces PaLM, a massive 540-billion parameter language model trained using its new Pathways system. The model achieves state-of-the-art few-shot performance across hundreds of tasks.
Google Research introduces the Pathways Language Model (PaLM), a dense decoder-only Transformer model containing 540 billion parameters. This massive AI system represents the first large-scale application of Google's new Pathways orchestration system, which allows a single model to generalize across multiple domains and tasks with high efficiency. By pushing the boundaries of model scale, Google aims to unlock emerging capabilities in language understanding and generation.
Training PaLM requires an unprecedented level of computational power, utilizing 6,144 chips spread across two Cloud TPU v4 Pods. This setup marks the largest TPU-based system configuration ever used for training. The system scales effectively by combining data parallelism at the Pod level with standard data and model parallelism within each individual Pod, surpassing the hardware limitations of previous large language models.
When evaluated on hundreds of language understanding and generation tasks, PaLM achieves state-of-the-art few-shot learning results by significant margins. As the scale of the model increases, its performance improves steadily across the board and reveals entirely new emergent capabilities. This breakthrough demonstrates that efficiently coordinating distributed computation across massive accelerator clusters yields substantial leaps in artificial intelligence performance.