AWS Scientist Wins ICLR Award for Hypercomplex Math Breakthrough
An AWS senior applied scientist and his collaborators earn an Outstanding Paper Award at ICLR 2021 for a technique that drastically reduces deep learning model parameters. Their method uses hypercomplex numbers to maintain effectiveness while cutting down the size of fully connected layers.
An Amazon Web Services senior applied scientist and his collaborators win an Outstanding Paper Award at the International Conference on Learning Representations (ICLR) 2021. The research, led by AWS scientist Aston Zhang alongside partners from Nanyang Technological University, ETH Zurich, and the University of Montreal, focuses on optimizing deep learning models. The paper explores how to significantly reduce the massive parameter counts required by neural networks without sacrificing their computational effectiveness.
Neural networks traditionally rely on fully connected layers that use standard matrix multiplication, which demands a high number of parameters. Zhang's team demonstrates that using quaternions, which are four-dimensional extensions of complex numbers, reduces the required parameters to just one-fourth of the traditional amount. Because quaternions possess four distinct components, they represent the operations of a fully connected layer much more efficiently than standard real-number matrices.
The award-winning research goes beyond simple quaternions by generalizing the math to accommodate higher-dimensional hypercomplex numbers with any number of imaginary components. This flexible mathematical representation yields even greater savings in parameter count and proves capable of capturing standard real-numbered operations as well. This breakthrough offers an extremely useful pathway for developers looking to build smaller, more efficient deep-learning models.