Researchers Teach Language Models to See for Common Sense AI
Researchers develop a technique called "vokenization" to give text-based AI models the ability to process images, bridging the gap between language and visual understanding.
Artificial intelligence models like GPT-3 produce remarkably human-like text but lack basic common sense because they train exclusively on written words. For example, a language model frequently guesses that sheep are black due to the familiar phrase "black sheep" rather than understanding visual reality.
Researchers at the University of North Carolina, Chapel Hill tackle this issue through a new method called "vokenization," which gives language models the ability to "see." While language models scale easily through unsupervised learning, computer vision systems learn directly from reality, making their combination highly valuable for creating AI that actually understands the physical world.
Merging these two AI disciplines requires building entirely new models from scratch using visual-language data sets that pair detailed image captions with pictures, rather than simply attaching an object recognition system to a text generator. This combined approach holds practical promise for future technologies like robotic assistants that need both visual navigation skills and human communication abilities.