NeurIPS 2020 Offers Practical Speed and Accuracy Tricks for Deep Learning
A review of NeurIPS 2020 highlights practical techniques for deep learning engineers, including a method that makes transformer pre-training 2.5 times faster.
The NeurIPS 2020 conference delivers highly practical insights for deep learning engineers who tackle real-world problems. By reviewing over 1900 paper titles and analyzing 175 abstracts, researchers identify key techniques that bridge the gap between theoretical AI research and practical engineering applications.
One standout paper introduces Switchable Transformers, a method that accelerates transformer pre-training by 2.5 times compared to standard models. This approach uses switchable gates to randomly skip certain layers during training, which saves 25 percent of computation time per sample and reduces the number of required training samples by 53 percent without sacrificing validation accuracy.
Another significant area of focus involves training neural networks robustly against noisy labels using mathematical techniques like weight Jacobians. Deep learning engineers benefit greatly from these empirical discoveries, as they provide immediate, actionable strategies to reduce computing costs and improve model reliability on downstream tasks.