Google Unveils PaLM, a 540-Billion Parameter Language Model
Google Research introduces PaLM, a massive 540-billion parameter language model trained using the new Pathways system. The model achieves state-of-the-art few-shot learning results across hundreds of tasks by efficiently scaling across thousands of chips.
Google Research introduces the Pathways Language Model (PaLM), a dense decoder-only Transformer model containing 540 billion parameters. This massive model trains efficiently using the new Pathways system, which orchestrates distributed computation across multiple TPU v4 Pods to realize Google's vision of a highly efficient, multi-task AI.
PaLM scales training to 6144 chips, marking the largest TPU-based system configuration used for training to date. By utilizing data parallelism at the Pod level across two Cloud TPU v4 Pods, the system significantly increases the scale of training compared to previous large language models that relied on smaller single-pod setups or GPU clusters.
Researchers evaluate PaLM on hundreds of language understanding and generation tasks, finding that it achieves state-of-the-art few-shot performance by significant margins. As the scale of the model increases, the performance improves across all evaluated tasks while simultaneously unlocking entirely new emergent capabilities.