Google Releases Open-Source Switch Transformer with 1.6 Trillion Parameters
Google Brain open-sources the Switch Transformer, a massive 1.6-trillion-parameter AI language model that trains up to seven times faster than its predecessor. The model achieves this speed by using a mixture-of-experts approach to keep computational costs low.
Researchers at Google Brain release the Switch Transformer, an open-source natural-language processing AI model that scales to 1.6 trillion parameters. This massive model improves training time by up to seven times compared to Google's previous state-of-the-art T5 model while maintaining comparable accuracy.
The system achieves these efficiency gains by utilizing a mixture-of-experts (MoE) paradigm that combines several Transformer attention blocks. Because only a specific subset of the model activates to process any given input, the total number of parameters increases dramatically without raising the computational cost per prediction.
This MoE approach builds on decades of machine learning research, including early work by deep-learning pioneer Geoff Hinton. By maximizing the parameter count while keeping the computing requirements constant, the Switch Transformer outperforms the T5-XXL model on standard language benchmarks despite training on only half the data.