OpenAI Codex Tackles Python Generation Using Repeated Sampling
A new GPT language model called Codex demonstrates strong Python writing capabilities by generating solutions from docstrings. The model achieves a high success rate through a strategy of repeated sampling.
Researchers introduce Codex, a GPT language model fine-tuned on public GitHub code to write Python programs. A distinct production version of this model currently powers the GitHub Copilot tool. To measure its functional correctness, the researchers release HumanEval, a new evaluation set that tests a model's ability to synthesize programs directly from docstrings.
Codex significantly outperforms previous models on this new benchmark by solving 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. The study reveals that repeated sampling is a surprisingly effective strategy for tackling difficult prompts. By generating 100 samples per problem, the model successfully solves 70.2% of the challenges.
Despite these impressive results, careful investigation exposes notable limitations in the system. Codex struggles specifically with docstrings that describe long chains of operations and with properly binding operations to variables. The paper also explores the broader impacts of deploying such powerful code generation technologies, discussing important implications for safety, security, and the global economy.