Self-Supervised Vision Transformers Reveal Surprising New Capabilities
Researchers introduce DINO, a self-supervised method that gives Vision Transformers unique properties like explicit semantic segmentation without labels. The approach achieves 80.1% top-1 accuracy on ImageNet using linear evaluation.
Researchers explore how self-supervised learning gives Vision Transformers (ViTs) unique properties that stand out compared to traditional convolutional networks. They find that self-supervised ViTs naturally contain explicit information about the semantic segmentation of an image, a characteristic that does not emerge as clearly with supervised ViTs or standard convnets. This discovery highlights a significant shift in how these models understand and process visual data without human-labeled annotations.
The study also reveals that these self-supervised ViT features serve as excellent k-NN classifiers, reaching an impressive 78.3% top-1 accuracy on ImageNet with just a small ViT model. To achieve these results, the researchers underline the critical importance of using a momentum encoder, multi-crop training strategies, and smaller image patches when working with ViT architectures.
Based on these findings, the team implements a simple self-supervised method called DINO, which functions as a form of self-distillation requiring no labels. The synergy between the DINO method and Vision Transformers proves highly effective, allowing the system to achieve an 80.1% top-1 accuracy on ImageNet in linear evaluation using a ViT-Base model.