Samples, effect sizes and "significant"
A result can be "statistically significant" and still too small to matter, and a small or self-selected sample can mislead. Learn the three numbers to check in any study.
PsychologyRead claims about the mindAbout 5 min: 3 reading, 2 practiceUpdated
Educational, not medical or mental-health advice.
- Judge whether a sample is big enough and like the people a claim is about
- Explain what "statistically significant" does and doesn't mean
- Use effect size to judge whether a result matters
Builds toward: Read claims about the mind. Judge a headline or a study about psychology before believing it or passing it on.
Read claims about the mind: lesson 3 of 5
- Read claims about the mind
- Linked is not the same as caused 6 min
- How experiments show cause 5 min
- Samples, effect sizes and "significant" 5 min
- What holds up: the replication crisis 7 min
- Popular psychology myths, checked 7 min
Try first
You haven't learned this yet, so guess. Guessing before you read helps the explanation stick. Nothing here is scored.
Already know this? Take a 3-question check and skip ahead.
Get all three right and the lesson is marked done.
Pairs with OpenStax, Psychology 2e, section 2.3 "Analyzing Findings"; the American Statistical Association's statement on p-values (Wasserstein and Lazar, 2016)
What it is
Three questions cover most of what you need to read the numbers in a study.
1. Who was studied, and how many? The sample is the people in the study. A small sample gives results that bounce around a lot by chance. And the sample should be like the people the claim is about: many psychology studies use university students, and an online poll only hears from people who chose to answer (a self-selected sample).
2. Is it "statistically significant"? This phrase means the result would be unlikely if there were really no effect at all, usually judged by a p-value below 0.05. It does not mean the effect is big, important or certain, and it is not the chance that the finding is true.
3. How big is the effect? The effect size says how large a difference is. Researchers often use Cohen's d, which measures a difference in standard deviations (a measure of how spread out scores are). As rough guides, Jacob Cohen suggested 0.2 is small, 0.5 medium and 0.8 large.
Quick check
Tap an answer to check it.
A study of 50,000 people finds a "statistically significant" difference in happiness between cat owners and dog owners of 0.3 points on a 100-point scale. What is the best reading?
- A tiny difference, and from a survey, so not proof of cause
Show answer
Unlikely if there were no real difference, but far too small to matter in daily life. With a huge sample, even a trivial difference becomes "significant". The effect size shows it is tiny.
Why it matters
Significant isn't the same as important. With a large enough sample, almost any difference passes the significance test, however small: significance on its own doesn't tell you the size. With a small sample, a real effect can be missed. That is why statisticians, in a 2016 statement from the American Statistical Association (Wasserstein and Lazar), warned that a p-value alone "does not measure the size of an effect or the importance of a result".
What a p-value is not. A p-value of 0.03 does not mean "there's a 3% chance the finding is wrong" or "a 97% chance it's true". It means: if there were no real effect, results at least this strong would turn up about 3% of the time.
Small samples swing. Imagine testing a memory trick on 10 people. One or two unusually good scorers can create a big apparent effect. Repeat it with another 10, and it may vanish. Small studies produce more of the dramatic results that make headlines, and more of the results that don't hold up.
Worked example
Headline: "Power naps boost exam scores by 20%." Open the study and check:
| Question | What the study says | Verdict |
|---|---|---|
| Who and how many? | 24 students at one university, who volunteered | Small and not typical of everyone |
| Significant? | p = 0.04 | Only just under the usual 0.05 line |
| How big? | d = 0.9; the "20%" is 4 extra points on a 20-point quiz: 20% of the maximum, not a 20% rise in scores | Large in this small sample, likely to shrink if repeated |
| Experiment? | Yes, randomly assigned to nap or rest | Can suggest cause, for students like these |
Fair summary: "A small experiment found that napping helped on a short quiz; it needs repeating with more people before we know how much it helps."
Common mistake
Reading "significant" in its everyday sense, meaning "big" or "important". In research it is a technical word about chance, not size. Always look for the effect size, or the actual numbers, next to it.
Key terms
- sample
- the people (or things) studied
- self-selected sample
- a sample of people who chose to take part, such as an online poll
- statistically significant
- a result that would be unlikely if there were no real effect, usually p below 0.05
- p-value
- how often results at least this strong would appear if there were no real effect
- effect size
- how big a difference or link is
- Cohen's d
- an effect size measuring a difference in standard deviations: about 0.2 small, 0.5 medium, 0.8 large
Practise the key terms
Tap a card to turn it over. 6 cards.
| Front | Back |
|---|---|
| sample | the people (or things) studied |
| self-selected sample | a sample of people who chose to take part, such as an online poll |
| statistically significant | a result that would be unlikely if there were no real effect, usually p below 0.05 |
| p-value | how often results at least this strong would appear if there were no real effect |
| effect size | how big a difference or link is |
| Cohen's d | an effect size measuring a difference in standard deviations: about 0.2 small, 0.5 medium, 0.8 large |
Practice
Quick quiz
Tap an answer to check it.
What does "statistically significant" mean?
- Significance is about chance, not size
- Significant results can still be wrong
Show answer
The result would be unlikely if there were no real effect. It is a statement about chance only; effect size tells you how big the result is.
A result has p = 0.02. Which statement is correct?
- A p-value doesn't give the chance a finding is true
Show answer
If there were no real effect, results this strong would appear about 2% of the time. The p-value assumes no effect and asks how surprising the data would be.
By Cohen's rough guide, an effect size of d = 0.2 is:
- Medium is about 0.5
- Large is about 0.8
Show answer
Small. Cohen's rough benchmarks are 0.2 small, 0.5 medium and 0.8 large.
An online poll of 30,000 readers finds most think a policy is a bad idea. What's the main problem?
- The size is fine; who answered is the problem
Show answer
The sample chose itself, so it may not represent people in general. Self-selected samples over-represent people with strong views or an interest in the topic.
Spot the error
One line is wrong. Tap it.
A friend summarises a study. Which line overstates the evidence?
- A plain description of the sample
- Correctly reported
- A plain description of the design
Show answer
"So the effect is huge and works for everyone.". Significance says nothing about size, and 30 adults can't show it works for everyone.
In short
- Check who was studied and how many: small or self-selected samples mislead.
- "Statistically significant" means the result would be unlikely if there were no real effect; it doesn't mean big or important.
- A p-value is not the chance a finding is true.
- Effect size shows how big a result is; Cohen's rough guide is 0.2 small, 0.5 medium, 0.8 large.
- Small studies produce more dramatic results, and more that don't hold up.
Mastery check
Answer without looking back. Score 80% or more and the lesson is marked done. Your result is kept only in this browser.
Check yourself
Answer in your head first, then open the answer. Nothing is saved.
Name the three questions to ask about a study's numbers.
Show answer
Who was studied and how many; whether the result is statistically significant; and how big the effect is.
Why can a huge study find a "significant" effect that doesn't matter?
Show answer
With a very large sample, even tiny differences become unlikely to be pure chance, so they pass the significance test while being too small to matter.
What is wrong with saying "p = 0.05 means a 5% chance the result is a fluke"?
Show answer
A p-value assumes there's no real effect and says how often results this strong would appear; it doesn't give the chance that this particular result is a fluke.
Further reading
- Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133.
- Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum.
- OpenStax, Psychology 2e, section 2.3, free online.
Finished? Mark it done to fill in your leaf on the Psychology map.