Meta Unveils OPT, an Open Source Alternative to GPT-3
Meta releases the OPT family of large language models to give researchers broader access to GPT-3-level technology. The release includes fully trained weights for models ranging from 125 million to 175 billion parameters.
Meta introduces the OPT family of transformer-based language models, offering nearly unrestricted access to source code and trained weights. The family includes eight models that range in size from 125 million to 175 billion parameters, giving researchers a powerful new tool to study natural language processing without relying on closed commercial APIs.
The flagship model, OPT-175B, mirrors the architecture and parameter count of OpenAI's GPT-3 and achieves comparable accuracy. Meta trains these models on publicly available datasets that include novels, news articles, Reddit posts, and a subset of The Pile, while also providing a detailed training log that documents the team's three-month development process.
While the massive 175 billion parameter version requires significant computing power and is restricted to non-commercial research by approved organizations, the seven smaller models are freely available to anyone. This release stands as the largest and most ambitious open-source language model to date, joining recent efforts by EleutherAI, Google, and Hugging Face's BigScience project to democratize large-scale AI.