Smaller AI Models Outperform GPT-3 Through Human Feedback Training
OpenAI researchers introduce InstructGPT, a language model trained with human feedback to better follow user instructions. Despite having 100 times fewer parameters, this aligned model consistently outperforms the massive GPT-3 in human evaluations.
Researchers at OpenAI address a major flaw in large language models where simply increasing the model size does not make them better at following a user's intent. Massive models like GPT-3 frequently generate outputs that are untruthful, toxic, or generally unhelpful because they lack proper alignment with human users. To solve this issue, the research team develops a new training method that relies heavily on human guidance to teach these AI systems exactly what users want.
The training process begins with supervised learning, where human labelers provide direct demonstrations of the desired model behavior based on various prompts. The team then collects a dataset of human-ranked model outputs and uses reinforcement learning from human feedback to further fine-tune the system. The researchers call the resulting aligned models InstructGPT, demonstrating that human input is a crucial ingredient for making artificial intelligence genuinely useful.
The results of this new approach are highly impressive, as the 1.3 billion parameter InstructGPT model actually outperforms the 175 billion parameter GPT-3 in human evaluations. In addition to producing much more helpful responses, InstructGPT shows significant improvements in truthfulness and generates far less toxic content. Even though the model still makes simple mistakes, this breakthrough proves that fine-tuning with human feedback is a highly promising direction for aligning language models with human intent.