Statistics Definitions > External Validity
External Validity Definition
External validity helps to answer the question: can the research be applied to the “real world”? If your research is applicable to other experiments, settings, people, and times, then external validity is high. If the research cannot be replicated in other situations, external validity is low. It’s important to know that your research is effective (internal validity) and that it is effective in other situations.
Historically, researchers have focused on internal validity. The scientific rigor of randomized, controlled experiments was often thought to be more important than the generalizability of results. More recently, researchers have been aiming for research that is more generalizable outside the lab. However, this isn’t as easy as it seems. External validity is one of the most difficult types of validity to achieve. One reason for this is that steps to make external validity high often result in a lowering of internal validity. Another reason is the multitudes of hidden and confounding variables that can affect your experimental outcome.
Population Validity and Ecological Validity
Population validity and ecological validity are types of external validity.
- Population validity answers the question: how well can the research on a sample be generalized to the population as a whole?
- Ecological validity answers the question: are your study results generalizable across different settings?
Threats to External ValidityThreats to external validity compromise your confidence in stating that your study results are applicable to other situations. They are explanations of how you might be wrong in making generalizations. For example, your conclusion might be incorrect, the changes in the dependent variable may not be due to changes in the independent variable, and variation in the dependent variable might be due to other causes. For example, extraneous variables may be competing with the independent variable to explain the study outcome.
Some specific examples of threats to external validity:
- Is your sample selected randomly? If not, it may be open to selection bias.
- Have you included a pretest? In some experiments, pretests may influence the outcome. A pretest might clue the subjects in about the ways they are expected to answer or behave.
- Are your participants taking multiple versions of the same test? If so, the practice effect might influence your results. For example the Wechsler Intelligence Scale for Children is highly influenced by the practice effect.
- Is your sample composed of a homogeneous population, like all low achievers or all high achievers? If so, your results probably won’t be generalizable to the “average” person.
- Are the results of your study tainted by the Hawthorne effect? Your study participants may be behaving differently because they know they are in an experimental study.
What is Replicability?
Replicability refers to whether the results from your test or experiment can be replicated if repeated exactly the same way. In order to demonstrate replicability, you must provide statistical evidence that shows your results can be used to predict outcomes in other experiments. An example of evidence that can suggest replicability is the probability value, or p-value. If your test produces a small p-value (usually less than .05 or 5 percent), it suggests your results are not due to chance. If your results can be replicated by different researchers in different experiments, then your results are called “robust.”
According to Science News, replicability is a big issue in science research. Around half of all pre-clinical research in the United States is not replicable, to a tune of $28 billion per year. In life sciences research, a major issue that causes experiments to be non-replicable is “faulty biological agents and reference materials, such as contaminated and misidentified cell lines.” The reality is, the tiniest of issues can lead to problems with replicating results.
Factors that Can Affect Replicability in Statistics
It isn’t just issues with the original experiment that can affect replicability. Other factors include:
- Single case studies and anecdotal evidence are not replicable as they are examples of a single instance of an event without corroborating evidence.
- Differences in experimental procedure in a subsequent experiment can affect replicability. For example, an inexperienced researcher or one using slightly different equipment may have trouble replicating results.
- People or companies who have a vested interest in confirming results from a test are more likely to find confirming evidence than those who do not have a vested interest.
- Results from replication attempts that are truly independent are more reliable than attempts that are not independent.