NeurIPS 2020 Reveals Self-Training Outperforms ImageNet Pre-Training in Vision Tasks
A deep analysis of NeurIPS 2020 computer vision papers shows that self-training consistently outperforms traditional ImageNet pre-training. This approach proves effective regardless of target dataset size or data augmentation strength.
A comprehensive review of NeurIPS 2020 computer vision papers reveals a major shift in how deep learning engineers approach model initialization. After analyzing hundreds of paper abstracts, researchers find that relying on ImageNet pre-trained weights is becoming less effective for modern computer vision tasks.
The research shows that the usefulness of ImageNet pre-training declines as the target dataset grows larger and data augmentation becomes stronger. In some cases, such as training on the COCO dataset for object detection, ImageNet pre-training actively hurts model performance rather than helping it.
Self-training emerges as a superior alternative, where a teacher model generates pseudo-labels to train a student model on additional data. This method consistently boosts performance across both low-data and high-data regimes, outperforming traditional pre-training regardless of augmentation strength, though it requires more computational resources.