Statistics How To

Shrinkage Estimator: Definition, Examples

Estimators > Shrinkage Estimator

What is Shrinkage?

shrinkage estimator

“Shrinking” scores by a percentage results in a better estimate of the true mean.


Shrinkage is where extreme values in a sample are “shrunk” towards a central value, like the sample mean. Shrinking data can result in:

However, the method has many disadvantages, including:

  • Serious errors if the population has an atypical mean. Knowing which means are “typical” and which are “atypical” can be difficult and sometimes impossible.
  • Shrunk estimators can become biased estimators, tending to underestimate the true population parameters.
  • Shrunk fitted models can perform more poorly on new data sets compared to the original data set used for fitting. Specifically, r-squared “shrinks.”

In Bayesian analysis, shrinkage is defined in terms of priors. Shrinkage is where:

“…the posterior estimate of the prior mean is shifted from the sample mean towards the prior mean” ~ Zhao et. al.

Models that include prior distributions can result in a great improvement in the accuracy of a shrunk estimator.

What is a Shrinkage Estimator?

A shrinkage estimator is a new estimate produced by shrinking a raw estimate (like the sample mean). For example, two extreme mean values can be combined to make one more centralized mean value; repeating this for all means in a sample will result in a revised sample mean that has “shrunk” towards the true population mean. Dozens of shrinkage estimators have been developed by various authors since Stein first introduced the idea in the 1950s. Popular ones include:

Other shrinkage methods include step-wise regression, which reduces the shrinkage factor to zero or one, least angle regression and cross-validatory approaches.

References:
Efron B, Morris C. Data analysis using Stein’s estimator and its generalisation. J. Am. Stat. Assoc. 1975;70:311–319

Horl, A.E., and Kennard, R.W. (1970). “Ridge Regression: Biased Estimation for Nonorthogonal Problems.” Technometrics. Vol. 12, No. 1, pp 55-68.

James, W., and Stein, C., (1961). “Estimation with Quadratic Loss.” Proceedings of the Fourth Berkeley Symposium, Vol. 1 (Berkeley, California: University of California Press), pp. 361-379.

Rolph, J.E., (1976). “Choosing…Estimators for Regression Problems.” Communication in Statistics- Theory and Methods. 5(9):789-802. January.

Stein C. (1956). “Inadmissibility of the usual estimator for the mean of a multivariate normal distribution“. Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability. Vol. 1. University of California Press; Berkeley, CA, USA: pp. 197–208

Zhao, Y., et. al (2010). “On Application of the Empirical Bayes…in Epidemiological Settings.” Int J Environ Res Public Health. 2010 Feb; 7(2): 380–394.
Published online Jan 28. doi: 10.3390/ijerph7020380.

------------------------------------------------------------------------------

If you prefer an online interactive environment to learn R and statistics, this free R Tutorial by Datacamp is a great way to get started. If you're are somewhat comfortable with R and are interested in going deeper into Statistics, try this Statistics with R track.

Comments are now closed for this post. Need help or want to post a correction? Please post a comment on our Facebook page and I'll do my best to help!
Shrinkage Estimator: Definition, Examples was last modified: October 12th, 2017 by Stephanie Glen