Apple Researchers Detail MM1, a State-of-the-Art Multimodal AI Model
A new research paper reveals key architectural and data strategies for building highly capable multimodal large language models, culminating in the MM1 family of AI models.
Researchers unveil MM1, a family of multimodal large language models that reaches up to 30 billion parameters and includes both dense and mixture-of-experts variants. Through comprehensive testing, the team discovers that the specific design of the vision-language connector plays a surprisingly negligible role in overall performance. Instead, the choice of image encoder, image resolution, and total image token count have a substantial impact on the model's capabilities.
The study highlights that the recipe for achieving state-of-the-art few-shot results relies heavily on a precise mixture of training data. By carefully combining image-caption pairs, interleaved image-text documents, and text-only data during large-scale pre-training, the models outperform other published pre-training results across multiple benchmarks. This strategic data mixing proves to be a crucial component for developing highly capable multimodal AI.
Thanks to this rigorous large-scale pre-training approach, MM1 exhibits highly appealing advanced properties. The model family demonstrates enhanced in-context learning abilities and sophisticated multi-image reasoning skills. These advancements enable MM1 to successfully perform few-shot chain-of-thought prompting, pushing the boundaries of what multimodal artificial intelligence can achieve.