Statistics Definitions > Practical Significance
Practical significance relates to whether a result from a statistical hypothesis test is useful in real life. It is a way to address some of the limitations with traditional testing and answers the question Do your results have real life applications and meaning?
Statistical significance is used in hypothesis testing to find out if test results are due to chance. Basically, hypothesis testing tells you whether there’s a difference between two sets of data (i.e. one set is bigger than the other). This isn’t of much practical use, because the real question is: how large is this difference? Imagine taking a potentially toxic chemotherapy drug because it’s “a bit better” than aspirin. Being told drug A is “better” than drug B isn’t enough information, but that’s exactly what you get with traditional testing.
Another problem with traditional significance testing is that although it can be calculated to a very high degree of accuracy (i.e. 99%), you could end up with statistically significant results when in fact none exist. John Tukey (1991) wrote, “It is foolish to ask ‘Are the effects of A and B different?’ They are always different—for some decimal place.” Despite a plethora of published studies that show “significant” differences between item A and item B, a large percentage can’t be replicated and therefore have little practical use in real life.
Practical Significance in Clinical Trials
Let’s say a clinical trial concludes that a new arthritis drug is effective for combating a disease, with 99% certainty. However, let’s say that the study researchers repeated the tests 100 times on different groups; This would almost certainly result in one or more of the groups showing improvement, which might lead the researchers to conclude that the drug looks promising. As well as the murky testing techniques, a single clinical trial, which might have built in biases as well as a host of other issues, so you wouldn’t say that the results from the trial have any practical significance. The best you could say was that the promising results indicate that the drug should be investigated further.
In fact, drug studies go through many different phases, with potentially many thousands of different studies, before the drug is released to market— giving it practical significance.
Assessing Practical Significance
Statistical significance is just one piece of the puzzle when it comes to assessing the practicality of trial results. In order to assess practical significance, you would also want to know the effect size, strength of any relationship (through a correlation coefficient), and confidence intervals.
That said, you would want to be careful not to “sanctify” any results (e.g. an effect size of .8), because that’s what led to the problems with traditional testing in the first place.
References
Kirk, R. Practical Significance: A Concept Whose Time Has Come. Retrieved March 8, 2019 from: https://www.researchgate.net/profile/Roger_Kirk/publication/238299074_Practical_Significance_A_Concept_Whose_Time_Has_Come/links/576ec7e908ae842225a88384/Practical-Significance-A-Concept-Whose-Time-Has-Come?origin=publication_detail
LeBlanc, M. Practical Significance. Retrieved March 8, 2019 from: www.oswego.edu/~leblanc/data/pracsig.ppt
Maarsman, M. Tukey’s Lament. Retrieved March 8, 2019 from: https://jasp-stats.org/2017/09/14/tukeys-lament-Lstatisticians-living-lie/
Tukey, J. W. (1991). The philosophy of multiple comparisons. Statistical Science, 6, 100-116.