Google Unveils PaLM: 540-Billion Parameter AI Model Powered by Pathways
Google Research introduces PaLM, a massive 540-billion parameter language model that achieves state-of-the-art few-shot learning performance. The model relies on the new Pathways system to efficiently train across thousands of advanced chips.
Google Research unveils the Pathways Language Model (PaLM), a dense decoder-only Transformer model featuring 540 billion parameters. This massive AI system achieves state-of-the-art few-shot performance across hundreds of language understanding and generation tasks, often beating previous models by significant margins. As the model scale increases, it not only improves overall performance but also unlocks entirely new capabilities.
The development of PaLM marks the first large-scale implementation of Google's Pathways system, which orchestrates distributed computation across multiple hardware accelerators. By utilizing this system, researchers efficiently train the single model across two Cloud TPU v4 Pods. This setup scales the training process to 6144 chips, making it the largest TPU-based system configuration ever used for training to date.
PaLM builds upon the success of previous large language models like GPT-3, GLaM, and LaMDA by pushing the limits of model scale and dataset diversity. Unlike earlier systems that relied on a single TPU v3 Pod or complex GPU cluster pipelines, PaLM uses data parallelism at the Pod level alongside standard data and model parallelism within each Pod. This approach demonstrates a major step toward Google's ultimate vision of creating a single, highly efficient model that generalizes seamlessly across multiple domains and tasks.