In other words, it’s a term that estimates your unknown parameter in a way that you lose the least amount of accuracy (as compared with having used the true value of that parameter).
Bayes Estimators & The Loss Function
Your data may be able to be represented by the function f(x|θ), where θ is a prior distribution. However, you don’t know the actual value of θ, so you have to estimate it. An estimator of θ is a real valued function δ(X1… Xn), not to be confused with Δx in calculus, which means a small change.
The loss function L(θ, a) where a ε R, is also a real valued function of θ. Our estimate here is a, and L(θ, a) tells us how much we lose by using a as an estimate when the true, real value of a parameter is θ.
There are different possible loss functions. For instance, the squared error loss function is given by L(θ a) = (θ –a)2. The absolute error loss function would be L(θ a) = |θ –a|.
In the first definition of Bayesian estimator at the beginning of this page, we said it was an estimator that minimized expected loss. That loss is represented by a loss function like one of those we’ve just described.
We can find the minimum expected loss by integrating. For a given X = x, the expected loss (E) is:
In this formula the Ω is the range over which θ is defined. p(θ | x) is the likelihood function; the prior distribution for the parameter θ over observations x. Call a * (x) the point where we reach the minimum expected loss. Then, for a*(x) = δ*(x), δ*(x) is the Bayesian estimate of θ.
Bayesian Estimator: References
Brynjarsdottir, Jenny. STAT 611 Lecture Notes: Lecture 12, Estimation.
Retrieved from https://www2.stat.duke.edu/courses/Fall12/sta611/Lecture12.pdf on March 9, 2018
Shiryaev, A. N. Bayesian estimator, Encyclopedia of Mathematics. Retrieved from : http://www.encyclopediaofmath.org/index.php?title=Bayesian_estimator&oldid=19043 on March 4, 2018.
Zhu, Wei. Bayesian Inference for the Normal Decision. Retrieved from http://www.ams.sunysb.edu/~zhu/ams571/Bayesian_Normal_wide.pdf on March 4, 2018