NeurIPS 2020 Retrospective Highlights Model Tuning and Training Tricks
A look back at the virtual NeurIPS 2020 conference reveals that proper model tuning and practical training tricks often outperform complex algorithmic research. Key takeaways include curriculum learning methods and the power of well-optimized baselines.
An impressive 23,000 attendees join the 2020 Conference on Neural Information Processing Systems virtually, making it a massive gathering for the machine learning community. The conference features a wide array of fascinating topics such as model understanding, self-supervised learning for audio, and the role of indigenous people in AI. One major takeaway from the event is that careful setup and proper tuning of models often yield significant performance improvements over complex new algorithms.
A prime example of this tuning philosophy shows how a well-optimized baseline achieves state-of-the-art results on the Movielens 10M dataset, easily beating years of specialized research. Because proper tuning remains a difficult task, researchers highly value "bag of tricks" papers that outline effective techniques for specific architectures like CNNs. These practical collections of tips give data scientists reliable methods to squeeze better performance out of existing models without inventing new mathematics.
One promising technique emerging from the conference is "Curriculum by Smoothing," which applies the human concept of progressive learning to neural network training. This method reduces high-frequency details in images during early training phases, allowing the model to learn basic shapes and structures before focusing on fine details. By mimicking how human vision naturally de-emphasizes high frequencies, this smoothing approach helps CNNs learn more effectively and steadily over time.