Google Open-Sources Trillion-Parameter Switch Transformer AI Model
Google Brain releases the Switch Transformer, a 1.6-trillion-parameter NLP model that trains seven times faster than its T5 predecessor. The model achieves this efficiency by using a mixture-of-experts approach to activate only a portion of its massive network for any given input.
Researchers at Google Brain open-source the Switch Transformer, a natural-language processing AI model that scales up to 1.6 trillion parameters. This massive model improves training time by up to seven times compared to Google's state-of-the-art T5 NLP model while maintaining comparable or better accuracy. The team details the architecture and performance of the model in a paper published on arXiv.
The Switch Transformer achieves these impressive efficiency gains by utilizing a mixture-of-experts (MoE) paradigm to combine several Transformer attention blocks. Because only a specific subset of the model activates to process any given input, researchers increase the total number of model parameters without increasing the computational cost. This approach allows the 1.6-trillion-parameter version to outperform the T5-XXL on pre-training perplexity metrics despite training on only half the data.
This development builds on a long history of MoE research that dates back to 1991 and includes deep-learning pioneer Geoff Hinton as a co-creator. The Switch Transformer specifically employs a modified MoE algorithm called Switch Routing, which allows the model to maximize its parameter count while keeping the number of FLOPS per training example constant. This breakthrough reinforces the industry consensus that NLP performance scales predictably with the number of parameters, even when using relatively small amounts of training data.