New Visual AI Model Learns From Text Instead of Fixed Labels
Researchers introduce a visual model that learns by matching internet images with text captions, bypassing the need for predetermined object categories. The system achieves impressive zero-shot performance across dozens of computer vision tasks.
OpenAI researchers introduce a new visual model that learns image representations by predicting which text captions match which images. Unlike traditional computer vision systems that rely on a fixed set of predetermined object categories, this approach uses natural language as a flexible source of supervision. The team trains this model from scratch on a massive dataset of 400 million image and text pairs collected from the internet.
This training method allows natural language to directly reference learned visual concepts or describe entirely new ones. As a result, the model enables zero-shot transfer to downstream tasks without requiring any additional, task-specific labeled data. The researchers evaluate the system by benchmarking it on over 30 different existing computer vision datasets.
The results show that the model transfers effectively to most tasks and often matches fully supervised baselines. Most notably, it achieves the same accuracy as the original ResNet-50 on ImageNet in a zero-shot setting without using any of the 1.28 million training examples. The researchers make their code and pre-trained model weights publicly available to encourage further development.