Google Unveils PaLM: 540-Billion Parameter AI Model Powered by Pathways
Google Research introduces PaLM, a massive 540-billion parameter language model that achieves state-of-the-art few-shot performance. The model demonstrates the first large-scale use of the new Pathways system to orchestrate training across thousands of advanced chips.
Google Research introduces the Pathways Language Model (PaLM), a dense decoder-only Transformer model with 540 billion parameters. This massive AI system achieves state-of-the-art few-shot learning results across hundreds of language understanding and generation tasks. As the scale of the model increases, its performance improves significantly and unlocks entirely new capabilities.
PaLM represents the first large-scale application of Google's Pathways system, which orchestrates distributed computation across multiple hardware accelerators. The system efficiently trains the model using 6,144 chips spread across two Cloud TPU v4 Pods. This configuration marks the largest TPU-based system ever used for training, utilizing a combination of data parallelism at the Pod level and standard parallelism within each Pod.
This breakthrough builds upon recent advancements in large language models like GPT-3, GLaM, and LaMDA. By pushing the limits of model scale, Google aims to realize its vision of a single model that generalizes across domains and tasks while maintaining high efficiency. The research shows that continued scaling yields predictable performance improvements and emergent abilities that smaller models simply do not possess.