xAI Unveils Grok-1 as Largest Open Source Mixture-of-Experts AI Model
Elon Musk's xAI releases Grok-1, a massive 314-billion-parameter model, under the Apache 2.0 license. The base model lacks safety tuning and matches early GPT-3.5 benchmark performance.
Elon Musk's AI company xAI releases Grok-1 as the largest open source Mixture-of-Experts model to date, offering its architecture and weights under the Apache 2.0 license. The massive system contains 314 billion parameters but only activates 25 percent of its expert networks for each input token, sharing a structural approach with models like Mistral's Mixtral and GPT-4. xAI builds this technology from scratch using a custom software stack based on JAX and Rust.
The released version represents the raw base model from a pre-training phase completed in October 2023, meaning developers receive an unrefined system that lacks optimization for specific applications. In standard benchmarks, Grok-1 performs on par with GPT-3.5, a level that many open-source models currently achieve. The model processes a context length of 8,192 tokens and relies on training data that cuts off in the third quarter of 2023.
Because Grok-1 does not include tuning with human feedback, it likely lacks the safety guardrails found in comparable commercial AI systems. xAI provides no details regarding its training data beyond mentioning large volumes of text and offers no ethical or safety guidelines for usage. This open source release potentially serves as a strategic counter-move in Musk's ongoing lawsuit against OpenAI, directly challenging the closed-source policies of his former company.