Google Brain Unveils 1.6-Trillion Parameter Switch Transformer
Google Brain introduces the Switch Transformer, a massive 1.6-trillion parameter language model that achieves a four-times pretraining speedup over the T5-XXL baseline. The model keeps computational costs under control by using a simplified Mixture of Experts routing technique.
Google Brain researchers introduce the Switch Transformer, a massive language model that scales to an unprecedented 1.6 trillion parameters. Despite its outrageous size, the model effectively controls computational costs and achieves a four-times pretraining speedup over Google's previously largest model, the strongly tuned T5-XXL baseline.
The team accomplishes this feat by simplifying the Mixture of Experts (MoE) routing algorithm to efficiently combine data, model, and expert-parallelism. Unlike traditional deep learning models that reuse the exact same parameters for all inputs, this MoE approach activates only a subset of the neural network weights for each incoming example, introducing beneficial sparsity to the architecture.
A key contribution of the study is the simplified routing paradigm's ability to significantly reduce communication and computational costs. By routing inputs to only a single expert network rather than multiple experts, the Switch Transformer delivers state-of-the-art results in language modeling and machine translation benchmarks without requiring an astronomical computational budget.