Google Unveils 540-Billion Parameter PaLM in Ongoing AI Scaling Race
Google introduces its massive Pathways Language Model (PaLM) as big tech companies compete to build ever-larger AI systems. The model relies on billions of web pages and thousands of specialized computer chips to process and generate human language.
Google unveils its Pathways Language Model (PaLM), a massive AI system containing 540 billion parameters. This release continues the competitive trend among big tech companies to build ever-larger language models, easily surpassing OpenAI's GPT-3 and nearly doubling the size of DeepMind's Gopher. The central question driving this expensive upscaling is whether these language models keep improving as they grow bigger.
Creating a model like PaLM requires an enormous amount of digital text and physical computing power. Developers feed the system a mixture of 780 billion language segments drawn mostly from social media, webpages, and books, with smaller portions coming from code, Wikipedia, and news. To process this massive dataset, Google relies on thousands of specialized machine learning chips arranged in efficient beehive-shaped pods.
The system uses a Transformer learning model to analyze the ingredients and map out relationships as numerical parameters. As the parallel processing folds the data together within and across the computer pods, the AI builds a complex understanding of language. The process finally stops once the system reaches the staggering milestone of 540 billion parameters, resulting in a model ready to understand and generate text.