Google Unveils PaLM, a 540-Billion Parameter Language Model
Google Research introduces PaLM, a massive 540-billion parameter language model trained using its new Pathways system. The model achieves state-of-the-art few-shot learning results across hundreds of tasks.
Google Research introduces the Pathways Language Model (PaLM), a dense decoder-only Transformer model with 540 billion parameters. The model trains efficiently across multiple TPU v4 Pods using the new Pathways system, which orchestrates distributed computation to generalize across different domains and tasks.
PaLM demonstrates the first large-scale use of the Pathways system by scaling training to 6144 chips, marking the largest TPU-based system configuration used for training to date. The system scales using data parallelism at the Pod level across two Cloud TPU v4 Pods, while utilizing standard data and model parallelism within each individual Pod.
When evaluated on hundreds of language understanding and generation tasks, PaLM achieves state-of-the-art few-shot performance by significant margins in many cases. As the scale of the model increases, the performance improves across all tasks while simultaneously unlocking entirely new emergent capabilities.