What is Regularized Regression?
Regularized regression is a type of regression where the coefficient estimates are constrained to zero. The magnitude (size) of coefficients, as well as the magnitude of the error term, are penalized. Complex models are discouraged, primarily to avoid overfitting.
“Regularization” is a way to give a penalty to certain models (usually overly complex ones). Ridge regression belongs to the set of L2 regularization tools. L2 regularization adds a penalty called an L2 penalty, which is the same as the square of the magnitude of coefficients. All coefficients are shrunk by the same factor, so all the coefficients remain in the model. The strength of the penalty term is controlled by a tuning parameter. When this tuning parameter (λ) is set to zero, ridge regression equals least squares regression. If λ = ∞, all coefficients are shrunk to zero. The ideal penalty is therefore somewhere in between 0 and ∞.
The other type of regularization, L1 regularization, limits the size of the coefficients by adding an L1 penalty equal to the absolute value of the magnitude of coefficients. This sometimes results in the elimination of some coefficients altogether, which can result in sparse models.
Types of Regularized Regression
Two commonly used types of regularized regression methods are ridge regression and lasso regression.
- Ridge regression is a way to create a parsimonious model when the number of predictor variables in a set exceeds the number of observations, or when a data set has multicollinearity (correlations between predictor variables).
- Lasso regression is a type of linear regression that uses shrinkage. Shrinkage is where data values are shrunk towards a central point, like the mean. This type is very useful when you have high levels of muticollinearity or when you want to automate certain parts of model selection, like variable selection/parameter elimination.
Dorugade and D. N. Kashid. Alternative Method for Choosing Ridge Parameter for Regression. Applied Mathematical Sciences, Vol. 4, 2010, no. 9, 447 – 456. Retrieved July 29, 2017 from: http://www.m-hikari.com/ams/ams-2010/ams-9-12-2010/dorugadeAMS9-12-2010.pdf.
Wessel N. van Wieringen. Lecture notes on RR. Retrieved July 29, 2017 from: https://arxiv.org/pdf/1509.09169.pdf