Meta Releases Open-Source OPT Models to Rival GPT-3
Researchers introduce OPT, a suite of open pre-trained transformer language models that match GPT-3's performance while significantly reducing carbon emissions. The release includes full model weights and code to help the research community study large language models.
Researchers from Meta introduce Open Pre-trained Transformers (OPT), a comprehensive suite of decoder-only pre-trained transformer language models. The collection ranges from 125 million to 175 billion parameters, giving the research community access to highly capable AI models that previously remained locked behind proprietary APIs.
The flagship OPT-175B model demonstrates performance that is directly comparable to GPT-3, but it requires only one-seventh of the carbon footprint to train. This massive reduction in environmental impact makes large-scale AI research much more sustainable without sacrificing the impressive zero-shot and few-shot learning capabilities that define modern language models.
To support further scientific inquiry, the developers fully and responsibly share the model weights alongside the necessary experimentation code. They also publish a detailed logbook that documents the specific infrastructure challenges encountered during the training process, providing valuable insights for other teams attempting to build similarly massive machine learning systems.