Regression Analysis > Weighted Least Squares
What is Weighted Least Squares?
Weighted Least Squares is an extension of Ordinary Least Squares regression that attaches non-negative constants (weights) to data points. It is used when:
- Your data violates the assumption of heteroscedasticity. Or,
- You want to focus your regression in certain areas (like low input values under a certain threshold). Or,
- You’re running the procedure as part of logistic regression or some other nonlinear function. Or,
- You have any other situation where data points should not be treated equally. For example, you might give more preference to points you know have been precisely measured and a lower preference to points that are estimated.
Instead of minimizing the residual sum of squares (as seen in OLS):
You minimize the weighted sum of squares:
A special case of weighted least squares is ordinary least squares; with OLS, all the weights are equal to 1. Therefore, solving the WSS formula is algebraically similar to solving the OLS formula.
Advantages and Disadvantages
Weighted least squares has several advantages over other methods, including:
- It’s well suited to extracting maximum information from small data sets.
- It is the only method that can be used for data points of varying quality.
- Requires that you know exactly what the weights are. Estimating weights can have unpredictable results, especially when dealing with small samples. Therefore, the technique should only be used when your weight estimates are fairly precise. In practice, precision of weight estimates usually isn’t possible.
- Sensitivity to outliers is a problem. A rogue outlier given an inappropriate weight could dramatically skew your results.
WLS can only be used in the rare cases where you know what the weight estimates are for each data point. When heteroscedasticity is a problem, it’s far more common to run OLS instead, using a difference variance estimator. For example, White (1980) suggests replacing S2(X’X)-1 by X’DX. This is a consistent estimator for X’ΩX:
While White’s consistent estimator doesn’t require heteroscedasticity, it isn’t a very efficient strategy. However, if you don’t know the weights for your data, it may be your best choice. For a full explanation of how to implement White’s consistent estimator, you can read White’s original 1908 paper for free here.
White, Halbert (1980). “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity”. Econometrica. 48 (4): 817–838. doi:10.2307/1912934