OpenAI Unveils Codex, a 12 Billion Parameter AI for Generating Code
OpenAI introduces Codex, a fine-tuned GPT-3 model that reliably writes Python code from plain English descriptions. The system solves a vast majority of programming problems in a new benchmark test designed to measure actual code execution rather than text similarity.
OpenAI announces Codex, a 12 billion parameter AI model that generates program code directly from natural language descriptions. Built upon the GPT-3 language model, Codex undergoes fine-tuning using 159 GB of publicly available Python code sourced from millions of GitHub repositories.
To accurately measure the system's capabilities, the OpenAI team creates HumanEval, a new benchmark dataset containing 164 programming challenges with corresponding unit tests. While the base GPT-3 model fails to solve any of these problems, Codex passes the unit tests for 28.7% of challenges on its first attempt and achieves a 77.5% success rate when generating 100 possible solutions per problem.
Unlike previous code generation models that rely on fuzzy text matching metrics like BLEU scores, Codex faces rigorous evaluation based on whether its output actually compiles and runs correctly. The researchers also adapt the text tokenizer to better handle the unique whitespace requirements of Python, paving the way for more accurate AI programming assistants like GitHub Copilot.