Google Unveils 540-Billion Parameter PaLM Language Model
Google Research introduces PaLM, a massive 540-billion parameter language model trained using its new Pathways system. The model achieves state-of-the-art few-shot performance across hundreds of language tasks.
Google Research introduces the Pathways Language Model (PaLM), a dense decoder-only Transformer model with 540 billion parameters. This massive model trains efficiently using the new Pathways system, which orchestrates distributed computation across multiple TPU v4 Pods. PaLM represents a major milestone in Google's goal to create a single model that generalizes across many different domains and tasks.
PaLM demonstrates the first large-scale application of the Pathways system, scaling training to 6,144 chips to set a new record for TPU-based system configurations. The training process uses data parallelism at the Pod level across two Cloud TPU v4 Pods, alongside standard data and model parallelism within each individual Pod. This approach provides a significant increase in scale compared to previous large language models.
Researchers evaluate PaLM on hundreds of language understanding and generation tasks, finding that it achieves state-of-the-art few-shot learning performance by significant margins. As the scale of the model increases, the performance improves across all tasks and unlocks entirely new emergent capabilities. These results show that pushing the limits of model scale yields major breakthroughs in artificial intelligence.