Statistics How To

Holm-Bonferroni Method: Step by Step

Familywise Error Rates > Holm-Bonferroni Method

You may want to read this article first: Familywise Error Rates.

What is the Holm-Bonferroni Method?

The Holm-Bonferroni Method (also called Holm’s Sequential Bonferroni Procedure) is a way to deal with familywise error rates (FWER) for multiple hypothesis tests. It is a modification of the Bonferroni correction. The Bonferroni correction reduces the possibility of getting a statistically significant result (i.e. a Type I error) when performing multiple tests. Although the Bonferroni is simple to calculate, it suffers from a lack of statistical power. The Holm-Bonferroni method is also fairly simple to calculate, but it is more powerful than the single-step Bonferroni.

Formula

The formula to calculate the Holm-Bonferroni is:


Where:

  • Target alpha level = overall alpha level (usually .05),
  • n = number of tests.

This next example shows how the formula works.

Example

Question: Use the Holm-Bonferroni method to test the following four hypotheses and their associated p-values at an alpha level of .05:

  • H1 = 0.01.
  • H2 = 0.04
  • H3 = 0.03
  • H4 = 0.005

Note: we already know the p-values associated with each hypothesis. If you don’t know the p-values, run a test for each hypothesis before attempting to adjust FWER using the Holm-Bonferroni method.

Step 1: Order the p-values from smallest to greatest:

  • H4 = 0.005
  • H1 = 0.01
  • H3 = 0.03
  • H2 = 0.04

Step 2: Work the Holm-Bonferroni formula for the first rank:
HB = Target α / (n – rank + 1)
HB = .05 / 4 – 1 + 1 = .05 / 4 = .0125.

Step 3: Compare the first-ranked (smallest) p-value from Step 1 to the alpha level calculated in Step 2:
Smallest p-value, in Step 1 (H4 = 0.005) < Alpha level in Step 2 (.125).
If the p-value is smaller, reject the null hypothesis for this individual test.

The p-value of .005 is less than .125, so the null hypothesis for H4 is rejected.

Step 4: Repeat the HB formula for the second rank .
HB = Target α / (n – rank + 1)
HB = .05 / 4 – 2 + 1 = .05 / 3 = .0167

Step 5: Compare the result from the HB formula in Step 4 to the second-ranked p-value:
Second ranked p-value, in Step 1 (H1 = 0.01) < Alpha level in Step 2 (.0167).
The p-value of .01 is less than .0167, so the null hypothesis for H1 is rejected as well.

Step 6: Repeat the HB formula for the third rank.
HB = Target α / (n – rank + 1)
HB = .05 / 4 – 3 + 1 = .05 / 2 = .025

Step 7: Compare the result from the HB formula in Step 6 to the third-ranked p-value:
Third ranked p-value, in Step 1 (H3 = 0.03) > Alpha level in Step 6 (.025).
The p-value of .03 is greater than .025, so the null hypothesis for H3 is not rejected.

The testing stops when you reach the first non-rejected hypothesis. All subsequent hypotheses are non-significant (i.e. not rejected).

Reference:
Holm, S. 1979. A simple sequential rejective multiple test procedure. Scandinavian Journal of Statistics 6:65-70

If you prefer an online interactive environment to learn R and statistics, this free R Tutorial by Datacamp is a great way to get started. If you're are somewhat comfortable with R and are interested in going deeper into Statistics, try this Statistics with R track.

Comments are now closed for this post. Need help or want to post a correction? Please post a comment on our Facebook page and I'll do my best to help!
Holm-Bonferroni Method: Step by Step was last modified: October 12th, 2017 by Andale

2 thoughts on “Holm-Bonferroni Method: Step by Step

  1. Hudson Nash

    Hello! There seem to be very few resources on Holm-Bonferroni correction, so this has been a great help.
    I am confused about why the “Target p-value” is equal to .05. Is this actually the target alpha?

  2. Andale Post author

    Thanks for your comment and I’m glad you found it helpful. Yes, it’s the target alpha (I changed it).
    Regards,
    S