Google Releases Gemma 2 Open Source Models That Rival Larger AI
Google DeepMind open-sources Gemma 2, a family of small language models that use knowledge distillation to outperform competitors of similar size and rival models twice as large.
Google DeepMind releases Gemma 2, the next generation of its open-source small language model family. The new release includes models with two billion, nine billion, and 27 billion parameters, all designed to deliver state-of-the-art performance while remaining accessible to developers. Google uses knowledge distillation to train the smaller models, a technique that leverages a larger teacher model to produce superior results over traditional next-token prediction methods.
The Gemma 2 architecture incorporates key advancements from Google's flagship Gemini model, such as Grouped-Query Attention (GQA) and a mix of global and local sliding window attention. These architectural upgrades allow the 27 billion parameter model to outperform similarly sized competitors like Qwen1.5 32B on major benchmarks including MMLU and GSM8K. Additionally, the model performs only a few percentage points below the significantly larger Llama 3 70B model.
This release continues the tech industry's trend of creating highly efficient, openly available language models that punch above their weight class. In human evaluations on the Chatbot Arena, the instruction-tuned 27B Gemma 2 model currently ranks as the highest-scoring open model, even edging out the massive Llama 3 70B. Google hopes this release unlocks advanced AI capabilities for the broader community and sparks future waves of research and development.