Researchers Develop Continuous Depth Neural Networks Using Differential Equations
A new family of deep neural network models replaces discrete hidden layers with continuous dynamics parameterized by differential equations. This approach offers constant memory cost and allows models to adapt their evaluation strategy to each individual input.
Researchers introduce a new family of deep neural network models that replace discrete sequences of hidden layers with continuous-depth architectures. Instead of defining specific layer transitions, these models parameterize the derivative of the hidden state using a neural network and compute the final output through a black-box differential equation solver.
These continuous-depth models provide significant computational advantages, including constant memory cost regardless of network depth. They adapt their evaluation strategy to each input and allow developers to explicitly trade numerical precision for speed, making them highly flexible for various machine learning tasks.
The authors demonstrate these capabilities through continuous-depth residual networks and continuous-time latent variable models. They also introduce continuous normalizing flows, a generative model that trains via maximum likelihood without needing to partition or order data dimensions, and show how to scalably backpropagate through any ODE solver for end-to-end training.