Google Unveils PaLM, a 540-Billion Parameter Language Model
Google AI introduces PaLM, a massive 540-billion parameter language model trained using the new Pathways system. The model achieves state-of-the-art few-shot performance across hundreds of language, reasoning, and coding tasks.
Google AI introduces the Pathways Language Model (PaLM), a dense 540-billion parameter decoder-only Transformer model. Researchers train this massive system using the new Pathways infrastructure, which allows a single model to be efficiently trained across multiple TPU v4 Pods simultaneously. This advanced training setup enables the model to scale up without sacrificing performance or efficiency.
PaLM achieves state-of-the-art few-shot learning results across hundreds of language understanding and generation tasks. It outperforms previous large models like GPT-3, Gopher, and Megatron-Turing NLG on 28 out of 29 standard English NLP benchmarks. The model demonstrates groundbreaking capabilities in complex areas that require deep comprehension, including challenging reasoning and code generation tasks.
The training data for PaLM combines English and multilingual datasets sourced from high-quality web pages, books, Wikipedia, discussions, and GitHub code. To handle this diverse input effectively, the researchers create a unique "lossless" vocabulary that preserves all whitespace for coding, splits unknown Unicode characters into bytes, and separates numbers into individual digit tokens. This specialized approach allows PaLM to process and generate both natural language and computer code with high accuracy.