Descriptive Statistics > Funnel Plot
What is a Funnel Plot?
A funnel plot is a scatter plot of individual studies, their precision and results.
Funnel plots have the following characteristics:
- Each dot represents a single study.
- The y-axis is usually the standard error of the effect estimate. Larger studies with higher power are placed towards the top. Lower powered studies are placed towards the bottom. However, other measures could also be plotted (e.g. the reciprocal of the standard error, the reciprocal of the sample size, or variance of the estimated effect).
- The x-axis shows the result for the study, sometimes expressed as an odds ratio.
- The plot should ideally resemble a pyramid or inverted funnel, with scatter due to sampling variation. The shape is expected because the studies have a wide range of standard errors. If the standard errors were the same size, the studies would all fall on a horizontal line.
What Does an Asymmetrical Funnel Plot Mean?
Funnel plots can be used as a check for bias in meta-analysis results. Asymmetry is commonly equated with publication bias and other kinds of reporting bias. However, funnel plots are not a good way to investigate publication bias (Sedgwick). There can be a number of reasons for asymmetrical funnel plots (also called small study effects). Sterne et. al (2011) list a slew of reasons, which include, but aren’t limited to:
- Poor methodological design, including fraud or inadequate analysis.
- Reporting bias, including delayed publication and location bias, selective outcome reporting and selective analysis reporting. Can also include language bias (i.e. only including those studies written in your native language).
- Chance: 95% of studies will usually fall within the triangular region if there are no biases or heterogeneity present in the studies. One possibility to skew the shape is that the errant 5% might all fall in one particular area by chance alone. The “95%” rule is actually a probability, meaning that chance alone could cause a higher or lower percentage than 95%, causing an asymmetrical shape that’s actually not an indication of any bias at all. This is especially true if only a small number of studies are included in the meta analysis.
- Study Heterogeneity. If heterogeneity results in a correlation between study size and intervention effects, this will result in an asymmetrical funnel (Terrin et. al).
The decision about whether a funnel plot is symmetric or not shouldn’t be based only on visual cues. Tests for asymmetry are available (one such test is Egger’s test), but they should be interpreted with caution. They may not have statistical validity, typically have low power, and they may be challenging to interpret. For a list of recommendations to follow when using these tests, see Sterne et. al’s BMJ article.
Sedgwick, P. (2013). Meta-analyses: how to read a funnel plot. BMJ 2013;346:f1342 doi: 10.1136/bmj.f1342.
Sterne et. al. Research Methods & Reporting Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ 2011; 343 doi: http://dx.doi.org/10.1136/bmj.d4002
Terrin N, Schmid CH, Lau J. In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. Journal of Clinical Epidemiology 2005;58:894-901