OpenAI Details Codex, the Deep Learning Model Powering GitHub Copilot
OpenAI publishes a new paper explaining how it adapted GPT-3 into Codex, the specialized deep learning model behind GitHub's Copilot coding assistant. By switching to supervised learning and adding code to the training data, researchers significantly boost the model's programming capabilities.
OpenAI releases a new paper detailing Codex, the deep learning model that serves as the backbone for GitHub's Copilot tool. Because the original GPT-3 model lacks coding samples in its training data, it fails to solve programming tasks despite its massive 175 billion parameters. To overcome this limitation, OpenAI researchers take a different approach to create Codex.
Instead of relying on the unsupervised learning methods used for GPT-3, the team utilizes supervised learning to fine-tune Codex specifically for programming. This strategic shift allows the model to achieve a remarkable 37.7 percent boost in performance. Codex effectively tackles the specialized task of translating natural language function descriptions into accurate source code.
The development of Codex highlights the "no-free lunch" theorem in machine learning, showing that models designed for specific tasks outperform generalized systems. While Codex excels at code generation, its natural language processing capabilities do not reach the same high levels of accuracy. This research provides valuable insight into the current strengths and boundaries of applying deep learning to software development.