Effective Sample Size: Definition, Examples

Sampling > Effective Sample Size

What is an Effective Sample Size?

An effective sample size (sometimes called an adequate sample size) in a study is one that will find a statistically significant effect for a scientifically significant event. In other words, an effective sample size ensures that an important research question gets answered correctly. In order to achieve this, your sample must be the “right” size: neither too big nor too small. This is more of an art than a science.

Note: The term “effective sample size” is also used in the calculation of design effects and has a much narrower definition; specifically, it’s the sample size you would expect if you used simple random sampling. See: What is a Design Effect?

Effect Sizes

An effective sample size is partially dependent on what effect size you’re willing to work with. Let’s say you’re comparing two drugs A and B. You know that drug B is better than A. Drug B could be ten times better, or it could be slightly better. This variability (twice as much? ten times as much?) is what is called an effect size. In general, gains in your ability to detect effect size and increase statistical power (the “strength” of a test) is increases with larger sample sizes.

Having smaller effect sizes (i.e. the ability to detect smaller significant differences) is preferable. For example, drugs A and B might be for cholesterol, and drug B decreases cholesterol levels by 10 mg/dL. A sample size of 20 might detect changes in cholesterol of 20 mg/dL with a 90% power, while a sample size of 80 might detect changes of 10 mg/dL with the same power. The obvious choice would be to go with the test with the better effect size (the one that would detect smaller changes of 10 mg/dL). However, this comes with an important trade off: halving the value of an effect size will generally quadruple the sample size.

Cost and Time Factors

Larger sample sizes involve more cost and time. Studies usually have limited budgets, which means you may not be able to get an ideal effective sample size. If you are limited in your sample size, changes in other aspects of your study design, like changing your data collection method, may result in a smaller effect size with the same sample size.

A Note on Scientific Significance

You’ll also want to consider whether a small change (like a 10 mg/dL change in cholesterol) has any real practical benefit. Normal cholesterol levels are 200 mg/dL while “high” is anything above 240 mg/dL. A drug that lowers cholesterol by 50 mg/dL would therefore be scientifically significant, while one that lowers it by 10 mg/dL is not. Therefore, although a sample size of 80 results in the desired effect size, it isn’t going to be scientifically significant and is therefore not an effective sample size.