Double-Descent, Emergence and Generalization in Machine Learning

The remarkable ability of modern neural networks to generalize improves with increasing network capacity, even when the number of model parameters or effective degrees of freedom exceeds the number of training data points.   Here I use dynamical mean field theory to show, in a simple setting of linear regression, that this so-called "double descent" behavior is the outcome of a phase transition in the stochastic field theory describing the training process. I show how this phenomenon is an example of emergent behavior, and specifically calculate the phase transition behavior, emergent rigidity, universal scaling theory and the specific features that give rise to the good generalization performance of modern neural networks. Our results are distinct from earlier work, because we calculate the time-dependence specifically, not just the equilibrium solutions.  This is what enables us to identify the origin of the emergent behavior.

Work performed in collaboration with Chan Li.