Overparameterization
Machine Learning models have varying levels of complexity, depending on the chosen architecture. For example, a fully connected neural network with two layers and six neurons in each has 42 parameters:
In classical machine learning, there is a requirement that the number of parameters in the model should be fewer than the number of unique training samples. Or in other words, model complexity should be below the amount of training data, where model complexity is the short hand for the number of parameters.
To understand the reason for this requirement, we need to look at how ML models are trained. Model training just means optimizing the parameter values or weights of the model in such a manner that the output of the model and the desired output differ as little as possible. For example, we could give the model a state of a chess game, and the optimal output would be the best move. Or we could give the model and image and ideal output is whether there is a cat in the image or not.
Now if we just have a set of images with and without a cat, and we know which is which, we can start training our model. As we want our model to be as accurate as possible, we train models with varying complexity until we find one that can perfectly tell when there is a cat in an image. Great, we now have a cat-recognizing model.
But when we give the model an image it hasn’t seen before, it almost certainly fails at the task. The model is not any better at recognizing cats than just flipping a coin. This problem is known as overfitting and one of the causes is that if a model learns a training set too well, it does not generalize. So strangely, when we train the model we don’t want the results to be too good.
In our example, we tested models with varying complexity and chose the one that had the best training error. But what likely happened was that our model learned to memorize the training set. Instead of learning to identify features from the image that reveal cats, it learned a correspondence between specific samples and the desired output. The absolute worst-case scenario of overfitting happens when model complexity matches the amount of training data we have.
The worst-case scenario is typically identified with a validation set. The idea is simply to split the available data into two groups, where the model is trained on the first set. When the model is trained (its weights are optimized), we can test its prediction accuracy on the left-out validation data set. If the model generalizes well, it performs well on the validation set.
The classical way to improve generalization is to make sure model complexity is low enough to make memorization impossible. Then for the model to generalize, it needs to find a way to represent useful features for the given task.
But what if we keep increasing model complexity beyond the amount of data we have for training? The increase gives birth to an effect called double descent: When generalization is plotted against model complexity, it first improves but starts to degrade when approaching the interpolation threshold - the point where model complexity and amount of data match. But when we increase model complexity further, the generalization can improve beyond what we achieved in the low-complexity region.
This region is called overparameterization and it is very often used in contemporary machine learning. After the model has bandwidth to do more than just memorize the training set, it can start learning useful features again. And as the complexity is increased, we have many different models that perform equally well with the training set, but some of those models have better generalization. When we optimize the models in this region - when model complexity exceeds training data - we can get models with better generilization than models in the classical region.
The lesson for data dcientists is clear - we need to be mindful of model complexity and make decisions whether the model should be overparameterizised or not.
Plots and animation made with help from Claude Opus 5.5.