Statistical Conclusion Validity

Statistics Definitions > Statistical Conclusion Validity

You may find it helpful to read this article first: Reliability and Validity in Research.

What is Statistical Conclusion Validity?

Statistical Conclusion Validity(SCV), or just Conclusion Validity is a measure of how reasonable a research or experimental conclusion is. For example, let’s say you ran some research to find out if two years of preschool is more effective than one. Based on the data, you conclude that there’s a positive relationship between how well a child does in school and how many years of preschool they attended. Conclusion validity well tell you how reliable that conclusion is.


Conclusion validity is only concerned with the question: Based on the data, is there a relationship or isn’t there? It doesn’t delve into specifics (like reliability tests) about what kinds of relationship exist. It can be used for qualitative research as well as quantitative research. That said, if you use the term statistical conclusion validity, that’s usually taken as meaning there’s some type of statistical data analysis involves (i.e. that your research has quantitative data).

It’s important to realize that there’s no such thing as perfect validity. Type 1 errors and Type 2 errors are a part of any testing process, so you can never be 100% certain that your conclusions are correct. However, SCV refers to reasonable conclusions based on your data — not perfect ones.

Threats to Statistical Conclusion Validity

Threats lead you to make incorrect conclusions about relationships. They include:

• Fishing (mining the data and repeating tests to find something…anything! significant…): can result in incorrectly concluding there is a relationship when in fact there is not.
• Low statistical power can cause you to incorrectly conclude there is no relationship between your variables.
• Poor reliability of treatment implementation: if you haven’t used standard procedures and protocols, it could cause you to underestimate effects.
• Random irrelevancies in the setting: this means any distraction, from weather that’s too hot to dealing with cantankerous people.
• Restriction of range: can also lead to incorrect estimates.
• Unreliable measures: can result in over- or underestimating the size of the relationship between variables.
• Violated assumptions for tests: can cause a multitude of problems including overestimating or underestimating effects.

Other Types of Validity

Three other types of validity are used to analyze research and tests:

• External Validity: the test or research can be applied to other areas.
• Internal Validity: the test or instrument is measuring what it’s supposed to.
• Construct Validity: the research/tests are well-constructed using established standards and methods.