Mistral AI Releases Mixtral 8x7B to Outperform Llama 2 and Rival GPT-3.5
Mistral AI unveils Mixtral 8x7B, a sparse mixture-of-experts model that delivers top-tier performance at a fraction of the cost. The open-weight model matches GPT-3.5 on most benchmarks while running six times faster than Llama 2 70B.
Mistral AI releases Mixtral 8x7B, a high-quality sparse mixture-of-experts model with open weights under the Apache 2.0 license. This new model outperforms Llama 2 70B on most benchmarks and offers inference speeds that are six times faster. It stands as the strongest open-weight model available today, matching or even surpassing GPT-3.5 across standard evaluations while providing an exceptional cost-to-performance ratio.
The architecture relies on a decoder-only design where a router network selects two out of eight distinct parameter groups to process each token. Although Mixtral contains 46.7 billion total parameters, it actively uses only 12.9 billion parameters per token. This approach allows the model to handle inputs and generate outputs at the exact same speed and cost as a much smaller 12.9 billion parameter model.
Mixtral supports a large context window of 32,000 tokens and fluently processes English, French, Italian, German, and Spanish. It also demonstrates strong capabilities in code generation and achieves an impressive score of 8.3 on MT-Bench when fine-tuned for instruction following. By making this architecture openly available, Mistral AI aims to equip developers with the tools needed to foster new AI inventions and applications.