You may want to read this article first: What is a Sufficient Statistic?
What is the Rao-Blackwell Theorem?
The Rao-Blackwell theorem (sometimes called the Rao–Blackwell–Kolmogorov theorem or Rao-Blackwellization) is a way to improve the efficiency of initial estimators. Estimators are observable random variables used to estimate quantities. For example, the (observable) sample mean is an estimator for the (unknown) population mean.
More formally, the theorem is defined as follows (from Sahai & Ojeda):
Suppose g(X) is an unbiased estimator of a scalar parametric function h(θ) and Ti(X), i = 1,2,…,p, are jointly sufficient for (θ); then there exists an estimator u(T), depending on the data only through the sufficient statistics, such that E[u(T)] = h(θ) and Var[u(T)] ≤ Var[g(X)].
For a proof of the theorem, read Prof. Charles Elkan’s Intuition, Lemmas and Start of Proof.
Note: Although Rao-Blackwellization can improve an estimator, it doesn’t always produce a uniformly minimum variance unbiased estimator.
Sufficiency
Improvement in efficiency is obtained by taking the statistic’s conditional expectation with respect to a sufficient statistic (assuming one exists). A sufficient statistic is a statistic that summarizes all of the information in a sample about a chosen parameter. In other words, a statistic is “sufficient” if it retains all of the information about the population that was contained in the original data points. According to statistician Ronald Fisher, “…no other statistic that can be calculated from the same sample provides any additional information as to the value of the parameter.”
Rice (2006) states the basic idea is that “…if an estimator is not a function of a sufficient statistic, it can be improved.” The improved statistic is optimal by any measure, including the mean squared error criterion or any other standard.
Applications
The Rao-Blackwell theory has many applications, including estimation of prediction error and producing estimates from sample survey data. For example, observations in adaptive sampling are found in sequence; each new observation depends on one or more characteristics from prior observations. Improved estimators can also be found by taking an average of estimators over every possible order .
References
Elkan, C. Rao-Blackwell Theorem: Intuition, Lemmas and Start of Proof. Retrieved July 5, 2017 from: http://cseweb.ucsd.edu/~elkan/291winter2005/lect04.pdf
Fisher, R.A. (1922). “On the mathematical foundations of theoretical statistics”. Philosophical Transactions of the Royal Society A 222: 309–368.
John Rice (2006). Mathematical Statistics and Data Analysis, Page 3
Hardeo Sahai, Mario M. Ojeda. Analysis of Variance for Random Models, Volume 2: Unbalanced Data.