Design of Experiments > Ecological Validity
What is Ecological Validity?
Ecological validity answers the question: “are your study results generalizable across different settings?” In other words, if you took your study to different locations (i.e. outside the test setting), would you still get the same results? Sterile lab environments are helpful for controlling variables in an experiment, but the trade-off is the results can rarely be applied to the real world.
Ecological Validity is a specific type of external validity. External validity refers to your ability to generalize your experimental results across populations, places, and time; ecological validity is limited to how the experimental results apply to today’s society.
The Problems with Experimental Settings
- Lighting, noise levels, equipment malfunctions and other distractions are minimized or eliminated.
- Clear instructions are given to avoid potential confusion.
- Ergonomic furniture, controlled lighting and refreshments may be provided to avoid fatigue.
Some (or all) of these factors may not appear in a “real world” setting, which means that your experiment may not be repeatable in the real world. The goal is to limit how much the testing environment affects your results. It sounds simple, but in practice it’s hard to achieve for several reasons including:
- Raising external validity often results in a lower level of internal validity.
- Hidden and confounding variables can (and usually do) affect your experimental outcome. The more you try to raise ecological validity, the less control you will have over confounding variables in your experimental setting.
One solution is to perform experiments in a natural setting where people are unaware that an experiment in taking place. This can be impossible to achieve in some experiments. Let’s say you were studying how much of an airline’s safety instructions people remember during an emergency landing. You wouldn’t be able to simulate that situation exactly; in this case, it would be unethical to create an experiment with high ecological validity. But you could take steps to increase validity by making the setting as natural as possible. For example, you would get higher validity by performing the experiment in a plane simulator that you would getting people to watch a movie of a crash and have them pretend they’re in the movie.
How realistic you can make your setting is limited by ethical boundaries. One famous experiment that attempted to create a “real life” environment was the Stanford Prison experiment. In the study, a prison-like environment was created in a Stanford University basement for two weeks. Students were assigned the roles of “guard” or “prisoner.” Some of the “guards” behaved in a brutal and sadistic manner. Four “prisoners” had to leave the experiment early, suffering from potentially long-term mental anguish caused by the study. Although the study had many strengths, it did lead to the establishment of formal ethical guidelines by the American Psychological Association. An institutional review board (US) or ethics committee (UK) must review studies before they are undertaken. In addition, most institutions have a review board that looks at a study’s plans before approval.
The Ecological Validity Debate
The term “ecological validity” was originally coined by Egon Brunswick, who gave it a very narrow meaning with correlations in perceptual tasks. Specifically, it’s about how organisms use cues in the habitat to make conclusions. Over the decades, the term has been used and misused in many different settings. This has led to a debate over what the term actually means. In short, there is no clear consensus as to what “ecological validity” actually refers to; there are multiple definitions. Even the methods researchers use to try and improve ecological validity can contradict one another.
The idea that testing environment affects experimental results is probably the most popular definition, and the one I’ve chosen to focus on in this article. However, you should probably be aware there are other definitions out there. For example, a similar definition is how psychological phenomena in the real world manifests in experimental settings. Before beginning the process of testing for ecological validity, it would be wise to figure out what exactly your professor (or department, or publication) defines ecological validity as.
References:
Brunswik, E. (1956). Perception and the representative design of psychological experiments. (2nd ed.). Berkeley: University of California Press.
Fahi, S. et.al. On the Ecological Validity of a Password Study. Retrieved 2/21/2016 from CMU.EDU.