Statistics How To

Weighted Least Squares

Regression Analysis > Weighted Least Squares

What is Weighted Least Squares?

Weighted Least Squares is an extension of Ordinary Least Squares regression that attaches non-negative constants (weights) to data points. It is used when:

  1. Your data violates the assumption of heteroscedasticity. Or,
  2. You want to focus your regression in certain areas (like low input values under a certain threshold). Or,
  3. You’re running the procedure as part of logistic regression or some other nonlinear function. Or,
  4. You have any other situation where data points should not be treated equally. For example, you might give more preference to points you know have been precisely measured and a lower preference to points that are estimated.

Formula

Instead of minimizing the residual sum of squares (as seen in OLS):
weighted least squares


You minimize the weighted sum of squares:
wss2


A special case of weighted least squares is ordinary least squares; with OLS, all the weights are equal to 1. Therefore, solving the WSS formula is algebraically similar to solving the OLS formula.

Advantages and Disadvantages

Weighted least squares has several advantages over other methods, including:

  • It’s well suited to extracting maximum information from small data sets.
  • It is the only method that can be used for data points of varying quality.

Disadvantages include:

  • Requires that you know exactly what the weights are. Estimating weights can have unpredictable results, especially when dealing with small samples. Therefore, the technique should only be used when your weight estimates are fairly precise. In practice, precision of weight estimates usually isn’t possible.
  • Sensitivity to outliers is a problem. A rogue outlier given an inappropriate weight could dramatically skew your results.

Alternatives

WLS can only be used in the rare cases where you know what the weight estimates are for each data point. When heteroscedasticity is a problem, it’s far more common to run OLS instead, using a difference variance estimator. For example, White (1980) suggests replacing S2(X’X)-1 by X’DX. This is a consistent estimator for X’ΩX:
whites


While White’s consistent estimator doesn’t require heteroscedasticity, it isn’t a very efficient strategy. However, if you don’t know the weights for your data, it may be your best choice. For a full explanation of how to implement White’s consistent estimator, you can read White’s original 1908 paper for free here.

Reference:
White, Halbert (1980). “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity”. Econometrica. 48 (4): 817–838. doi:10.2307/1912934

If you prefer an online interactive environment to learn R and statistics, this free R Tutorial by Datacamp is a great way to get started. If you're are somewhat comfortable with R and are interested in going deeper into Statistics, try this Statistics with R track.

Comments are now closed for this post. Need help or want to post a correction? Please post a comment on our Facebook page and I'll do my best to help!
Weighted Least Squares was last modified: October 12th, 2017 by Andale