Google Unveils PaLM, a 540-Billion Parameter Language Model
Google Research introduces PaLM, a massive 540-billion parameter language model trained using its new Pathways system. The model achieves state-of-the-art few-shot learning performance across hundreds of tasks.
Google Research introduces the Pathways Language Model (PaLM), a dense decoder-only Transformer model with 540 billion parameters. The company trains this massive AI using the new Pathways system, which orchestrates distributed computation across multiple TPU v4 Pods. PaLM represents a major milestone in Google's vision to create a single model that generalizes efficiently across diverse domains and tasks.
The training process utilizes 6144 chips, marking the largest TPU-based system configuration used for training to date. Google scales the training using data parallelism at the Pod level across two Cloud TPU v4 Pods, combined with standard data and model parallelism within each Pod. This approach significantly surpasses the scale of previous large language models that relied on single TPU v3 Pods or smaller GPU clusters.
Evaluations across hundreds of language understanding and generation tasks reveal that PaLM achieves state-of-the-art few-shot performance. The model outperforms predecessors by significant margins in many cases, proving that increasing the scale of the model directly improves performance. Furthermore, pushing these limits of model scale unlocks entirely new emergent capabilities in language understanding.