OpenAI Codex Dominates July 2021 arXiv Hot Papers List
OpenAI's Codex, a GPT language model fine-tuned on code, tops the list of most tweeted arXiv papers from July 2021. The model significantly outperforms its predecessors in writing functional Python code.
The most tweeted papers uploaded to arXiv during July 2021 highlight major advancements in artificial intelligence, with OpenAI's Codex taking the top spot. Powered by data from the Arxiv Sanity Preserver, the monthly roundup shows that research focusing on the intersection of natural language processing and programming attracts massive attention from the tech community.
OpenAI introduces Codex as a GPT language model specifically fine-tuned on publicly available code from GitHub. A distinct production version of this model currently powers GitHub Copilot, demonstrating an immediate real-world application for this academic research. The paper details how the model approaches the task of synthesizing programs directly from docstrings.
To measure the model's capabilities, the researchers release HumanEval, a new evaluation set designed to test functional correctness in code generation. Results show that Codex solves 28.8% of the problems on its first attempt, while GPT-3 solves 0% and GPT-J solves 11.4%. By using a repeated sampling strategy of 100 samples per problem, the model successfully solves 70.2% of the presented challenges, although the researchers note distinct limitations with complex, multi-step operations.