Meta's Chameleon AI Surpasses GPT-4 in Mixed Image and Text Generation
Meta releases Chameleon, a mixed-modal AI model that processes and generates images and text using a single token-based architecture. The model outperforms GPT-4 and Gemini Pro in human evaluations.
Meta's FAIR team releases Chameleon, a mixed-modal AI model that understands and generates both text and image content seamlessly. Unlike traditional models that rely on separate encoders and decoders for different data types, Chameleon uses a single token-based representation to process mixed sequences end-to-end. In human evaluations, judges prefer Chameleon's output over GPT-4 in 51.6% of trials and over Gemini Pro in 60.4% of trials.
The researchers train two versions of the model, Chameleon-7B and Chameleon-34B, on over four trillion tokens of mixed text and image data before fine-tuning them for safety and alignment. By quantizing images into discrete tokens and training on mixed-modal data from scratch, the model learns to jointly reason over both modalities. This unified approach allows Chameleon to achieve state-of-the-art performance on visual question answering and image captioning benchmarks.
Scaling the model presents significant challenges, as training instabilities occur when the parameter count exceeds eight billion or the dataset surpasses one trillion tokens. To resolve these optimization and architectural design hurdles, the Meta team modifies the standard Transformer architecture, particularly regarding shared model weights across input modalities. These innovations enable stable training at larger scales and unlock entirely new possibilities for multimodal interaction.