OpenAI Codex Leads July's Most Discussed AI Research on arXiv
OpenAI's Codex, a GPT model trained on code that powers GitHub Copilot, tops the list of July 2021's most tweeted arXiv papers. The research demonstrates significant advancements in synthesizing functional Python programs from text prompts.
OpenAI's Codex dominates the list of the most shared arXiv papers from July 2021, according to data from the Arxiv Sanity Preserver. This highly discussed research introduces a GPT language model that undergoes fine-tuning on public code from GitHub, ultimately serving as the underlying engine for the GitHub Copilot tool.
The researchers evaluate Codex using HumanEval, a newly released benchmark designed to measure functional correctness in synthesizing programs directly from docstrings. Results show that Codex successfully solves 28.8% of these problems, a massive improvement over GPT-3's 0% and GPT-J's 11.4%, highlighting a major leap forward in artificial intelligence code generation.
The study also reveals that repeated sampling from the model acts as a surprisingly effective strategy for tackling difficult programming prompts. By generating 100 samples per problem, the system solves 70.2% of the challenges, though the authors note distinct limitations, such as struggles with docstrings that describe long chains of operations.