OpenAI Codex Dominates July 2021 arXiv Discussion
OpenAI's Codex model tops the list of most-discussed arXiv papers from July 2021. The code-generating system shows significant improvements over previous language models.
OpenAI's Codex model emerges as the most tweeted paper uploaded to arXiv during July 2021. The research introduces a GPT language model that undergoes fine-tuning on publicly available code from GitHub, showcasing impressive capabilities in writing Python code.
A distinct production version of this Codex model directly powers GitHub Copilot. To properly measure the system, the researchers release HumanEval, a new evaluation set designed to measure functional correctness for synthesizing programs directly from docstrings.
The results show Codex solves 28.8% of the problems on this new benchmark, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, the researchers find that repeated sampling from the model serves as a surprisingly effective strategy, allowing the system to solve 70.2% of problems when generating 100 samples per problem.