OpenAI Codex Outperforms Standard Language Models in Python Code Generation
Researchers introduce Codex, a GPT model fine-tuned on GitHub code that successfully solves nearly 29 percent of Python programming challenges. A distinct version of this model currently powers the GitHub Copilot coding assistant.
Researchers introduce Codex, a GPT language model fine-tuned on public GitHub code, to study its ability to write Python programs. A distinct production version of this model directly powers GitHub Copilot, bringing advanced AI capabilities directly into developer workflows.
To measure the model accurately, the team creates HumanEval, a new evaluation set that tests functional correctness by asking the AI to synthesize programs from docstrings. On this challenging benchmark, Codex solves 28.8% of the problems, significantly outperforming GPT-3 which solves 0% and GPT-J which solves 11.4%.
The study reveals that repeated sampling is a surprisingly effective method for finding working solutions to difficult programming prompts, allowing the model to solve 70.2% of problems when generating 100 samples per task. However, the researchers also note distinct limitations, highlighting that the model struggles with docstrings describing long chains of operations and with properly binding operations to variables.