xAI Releases Grok-1 Open-Weights Model for Public Download
xAI officially releases the Grok-1 open-weights model, a massive 314-billion parameter language model, on GitHub. Developers need significant GPU memory to run the Mixture-of-Experts architecture locally.
xAI releases the Grok-1 open-weights model on GitHub, providing JAX example code for developers to load and run the massive system locally. The model boasts an impressive 314 billion parameters and utilizes a Mixture of 8 Experts (MoE) architecture, activating 2 experts per token across its 64 layers.
Due to the enormous size of the model, users require a machine with substantial GPU memory to handle the test code. xAI openly notes that the current MoE layer implementation is intentionally not optimized for efficiency, as the team chose this approach to avoid custom kernels while validating the model's correctness.
The model supports several advanced features, including rotary embeddings, activation sharding, and 8-bit quantization, with a maximum context length of 8,192 tokens. Developers download the weights via a provided magnet link or directly through the Hugging Face Hub, and the entire release operates under the permissive Apache 2.0 license.