2020 Machine Learning Year in Review: Transformers, Ethics, and Self-Supervised Learning
A comprehensive review of 84 significant machine learning papers from 2020 highlights the dominance of Transformer models, advances in self-supervised learning, and the growing importance of AI ethics.
A new comprehensive review highlights 84 significant machine learning papers and articles published in 2020, categorizing them into 12 distinct sections ranging from image classification to real-world applications. The overarching theme of the year is the massive leap forward made by Transformer models, which establish new state-of-the-art results across various domains and fundamentally shift the landscape of artificial intelligence research.
In the realm of natural language processing, the massive GPT-3 model achieves unprecedented accuracy by leveraging enormous amounts of data and parameters. Transformers also break through in the image classification field, surpassing traditional CNN-based models on ImageNet, though this requires immense computational resources. Additionally, self-supervised learning techniques emerge as a major trend, achieving accuracy levels that rival traditional supervised learning methods without the need for manual data labeling.
Beyond pure performance metrics, the review points to critical developments in AI ethics and practical applications. Researchers introduce fractal image datasets to avoid copyright and discrimination issues inherent in traditional data like ImageNet. Meanwhile, the Deepfake problem spawns new detection methods using biometric signals, while also being utilized positively to protect victim privacy. Finally, the combination of machine learning with numerical simulation gains traction, allowing companies to run ultra-fast simulations by learning complex input and output patterns.