Statistics How To

Familywise Error Rate (Alpha Inflation): Definition

Post Hoc tests > Familywise Error Rate

What is the Familywise Error Rate?

The familywise error rate (FWE or FWER) is the probability of a coming to at least one false conclusion in a series of hypothesis tests . In other words, it’s the probability of making at least one Type I Error. The term “familywise” error rate comes from family of tests, which is the technical definition for a series of tests on data.

The FWER is also called alpha inflation or cumulative Type I error.

Formula

The formula to estimate the familywise error rate is:
FWE ≤ 1 – (1 – αIT)c
Where:

  • αIT = alpha level for an individual test (e.g. .05),
  • c = Number of comparisons.

For example, with an alpha level of 5% and a series of ten tests, the FWER is:
FWE = ≤ 1 – (1 – .05)10 = .401.
This means that the probability of a type I error is just over 40%, which is very high considering only ten tests were performed.

Controlling the FWER

familywise error rate

Some tests, especially in the sciences, can be rerun tens of thousands of times.

You need to control the FWER for one main reason: If you run enough hypothesis tests (dozens, hundreds, or sometimes tens of thousands) you’re highly likely to get at least one significant result — a “false alarm” where you incorrectly reject the null hypothesis.

Two main procedures are used to control the FWER: single step and sequential.

Single step

The single step procedure makes equal adjustments to each p-value. This keeps the overall alpha level at the desired level (e.g. .05) and is called a Bonferroni correction.

  1. Divide the alpha level by the number of tests you’re running and apply that alpha level to each individual test. For example, if your overall alpha level is .05 and you are running 5 tests, then each test will have an alpha level of .05/5 = .01.
  2. Apply the new alpha level to each test for finding p-values. In this example, the p-value would have to be .01 or less for statistical significance.

The Bonferroni correction has been criticized for (among other things) loss of power and a high probability of Type II errors.

Sequential

Similar to Bonferroni, but makes adaptive adjustments to each p-value. Several sequential methods exist. The easiest is probably the Holm-Bonferroni Method, but several others have been developed including the Sidak-Bonferroni and Holland-Copenhaver.

  1. Holm-Bonferroni: tests are run and then ordered from lowest to highest p-values. The individual tests are then tested (starting with the one with the lowest p-value) with an overall Bonferroni correction for all tests. See: Holm-Bonferroni Method for a step-by-step example.
  2. Sidak-Bonferroni (sometimes called the Boole or Dunn approximation): a variant of Bonferroni which uses a Taylor expansion (from calculus).

Reference:
Olejnik,S., Li, J., Supattathum, S., and Huberty, C.J. (1997). Multiple testing and statistical power with modified Bonferroni procedures. Journal
of educational and behavioral statistics, 22, 389-406.
Holland, B. S., and M. D. Copenhaver. 1987. An improved sequentially rejective Bonferroni test procedure. Biometrics 43: 417–423.

If you prefer an online interactive environment to learn R and statistics, this free R Tutorial by Datacamp is a great way to get started. If you're are somewhat comfortable with R and are interested in going deeper into Statistics, try this Statistics with R track.

Comments are now closed for this post. Need help or want to post a correction? Please post a comment on our Facebook page and I'll do my best to help!
Familywise Error Rate (Alpha Inflation): Definition was last modified: October 12th, 2017 by Andale

2 thoughts on “Familywise Error Rate (Alpha Inflation): Definition

  1. John Smith

    I’m a little loss at the description of criticism of the Bonferroni method. It is written that Bonferroni correction has been criticizing for loss of power and a high probability of Type II error. However, since you are reducing the alpha level, how is this a loss of power? Moreover, how does a loss of power lead to higher probability of Type II error. Maybe my understanding of power is incorrect, but shouldn’t higher power lead to higher Type II errors?

    Thank you in advance.

  2. Andale Post author

    “Since you are reducing the alpha level, how is this a loss of power?”
    Power is a function of the number of tests and corresponding critical value increases. See this image.
    “…how does a loss of power lead to higher probability of Type II error.”
    By definition, power is 1–β (type II error). So high values for beta mean lower power and vice versa.