What holds up: the replication crisis
When psychologists repeated 100 published studies, fewer than half gave the same result. Learn why, which famous findings fell, what changed, and how to tell a sturdy finding from a shaky one.
PsychologyRead claims about the mindAbout 7 min: 4 reading, 3 practiceUpdated
Educational, not medical or mental-health advice.
- Explain what replication is and what the 2015 project found
- Describe how p-hacking and publication bias produce false findings
- Use five signs to judge whether a finding is likely to hold up
Builds toward: Read claims about the mind. Judge a headline or a study about psychology before believing it or passing it on.
Read claims about the mind: lesson 4 of 5
- Read claims about the mind
- Linked is not the same as caused 6 min
- How experiments show cause 5 min
- Samples, effect sizes and "significant" 5 min
- What holds up: the replication crisis 7 min
- Popular psychology myths, checked 7 min
Try first
You haven't learned this yet, so guess. Guessing before you read helps the explanation stick. Nothing here is scored.
Already know this? Take a 3-question check and skip ahead.
Get all three right and the lesson is marked done.
Pairs with Open Science Collaboration (2015), "Estimating the reproducibility of psychological science"; OpenStax, Psychology 2e, section 2.3 "Analyzing Findings"
What it is
Replication means running a study again, with new participants, to see if the result comes out the same. A single study can be wrong for many reasons; a finding that replicates is much more trustworthy.
In 2015 a team of about 270 researchers, the Open Science Collaboration, repeated 100 studies published in three leading psychology journals. In the originals, 97% had found a statistically significant result. In the repeats, only 36% did, and the effects were on average about half as big. It became a central finding of what is called the replication crisis.
Quick check
Tap an answer to check it.
What did the 2015 project find when it repeated 100 psychology studies?
- Far fewer did: about a third
Show answer
About a third gave a significant result again, with effects about half as big on average. 97% of the originals were significant; only 36% of the repeats were.
Why it matters
How false findings get published.
- Publication bias: journals were more likely to publish exciting, significant results. Studies that found nothing often stayed in a drawer, so the published record looked more certain than the evidence was.
- P-hacking: researchers have many small choices: which measures to report, when to stop collecting data, which participants to leave out. Trying several and reporting the one that "worked" makes false results easy to find, without anyone meaning to cheat. Simmons, Nelson and Simonsohn (2011) showed that with such choices, a nonsense effect can be made significant far more often than the 5% you'd expect by chance.
- Small samples: as in the last lesson, small studies produce more flukes.
Famous findings that didn't hold up.
- Ego depletion, the idea that self-control is a resource that runs out like fuel. A large, coordinated repeat across 23 labs (Hagger and colleagues, 2016) found an effect close to zero.
- Power posing, the claim that standing in a "powerful" pose for two minutes changes your hormones and risk-taking. A larger repeat (Ranehill and colleagues, 2015) found no effect on hormones or risk-taking, only on how powerful people said they felt.
What changed. Many researchers now preregister studies: they publish their plan, measures and analysis before collecting data, so they can't quietly change them afterwards. Some journals accept registered reports, deciding to publish before the results are known. Samples are bigger, data are shared openly, and large multi-lab repeats are now common. Many classic findings, such as anchoring and the testing effect, have replicated well.
Worked example
You read: "Holding a warm drink makes you judge strangers as warmer people, study finds." How much should you trust it? Check five signs:
| Sign | Ask | Here |
|---|---|---|
| Replicated? | Has another team found it again? | Later direct repeats did not find it |
| Sample size | Dozens or thousands? | The original studies were small |
| Effect size | Plausible, or surprisingly big for a small nudge? | A big change in judgement from a brief, small touch |
| Preregistered? | Was the plan published first? | No, it predates the reforms |
| Fits other evidence? | Does it fit what we know? | It was surprising, and direct repeats did not support it |
Verdict: treat it as doubtful. A surprising, small, unrepeated finding is the kind most likely to fail.
Common mistake
Swinging from "a study proves it" to "psychology is all rubbish". Replication failures were more common in small, surprising studies. Large, repeated findings, such as the benefits of testing yourself and spacing your study, have held up again and again. The skill is telling them apart, not trusting or dismissing everything.
Key terms
- replication
- repeating a study with new participants to see if the result holds
- replication crisis
- the discovery, around 2011–2015, that many published findings failed to replicate
- publication bias
- exciting, significant results being more likely to be published than null results
- p-hacking
- trying different analyses or data choices until a result becomes significant
- preregistration
- publishing a study's plan and analysis before collecting data
- registered report
- a study accepted for publication on its plan, before the results are known
Practise the key terms
Tap a card to turn it over. 6 cards.
| Front | Back |
|---|---|
| replication | repeating a study with new participants to see if the result holds |
| replication crisis | the discovery, around 2011–2015, that many published findings failed to replicate |
| publication bias | exciting, significant results being more likely to be published than null results |
| p-hacking | trying different analyses or data choices until a result becomes significant |
| preregistration | publishing a study's plan and analysis before collecting data |
| registered report | a study accepted for publication on its plan, before the results are known |
Practice
Quick quiz
Tap an answer to check it.
In the 2015 project, what share of repeated studies found a significant result again?
- That was the share of the original studies
- Far fewer replicated
Show answer
About 36%. 97% of originals were significant, but only 36% of the repeats.
What is p-hacking?
- It's a research habit, not a crime against computers
- That's normal practice; p-hacking is how you get there
Show answer
Trying different analyses or data choices until a result becomes significant. Many small, flexible choices make false positives much more likely.
How does preregistration help?
- It does the opposite: it allows honest null results
Show answer
Researchers fix their plan before seeing the data, so they can't quietly change it to get a result. Deciding the analysis in advance removes the freedom that p-hacking relies on.
Which finding is most likely to hold up?
- Small, surprising and unrepeated is the shakiest kind
Show answer
One repeated by several large teams, with a moderate effect that fits other evidence. Replication, size and fit with other evidence are the signs of a sturdy finding.
Sort it
Which group does this belong to?
Repeated by independent teams
Replication is the strongest evidenceShow answer
Sign of a sturdy finding. Replication is the strongest evidence
A tiny nudge with a huge effect
Big effects from small nudges often shrink on repeatShow answer
Warning sign. Big effects from small nudges often shrink on repeat
Plan published before data collection
Preregistration limits p-hackingShow answer
Sign of a sturdy finding. Preregistration limits p-hacking
One study of 20 people, never repeated
Small and unrepeated results are the most likely to failShow answer
Warning sign. Small and unrepeated results are the most likely to fail
In short
- Replication means repeating a study to see if the result holds.
- In 2015, only 36% of 100 repeated psychology studies found a significant result again (97% of originals had), with effects about half as big.
- Publication bias, p-hacking and small samples produced many false findings.
- Ego depletion and power posing's effects on hormones and risk-taking did not hold up; testing yourself and anchoring did.
- Reforms: preregistration, registered reports, bigger samples, open data, multi-lab repeats.
Mastery check
Answer without looking back. Score 80% or more and the lesson is marked done. Your result is kept only in this browser.
Check yourself
Answer in your head first, then open the answer. Nothing is saved.
What is replication, and why does it matter?
Show answer
Repeating a study with new participants to see if the result holds; one study can be wrong, but a result that replicates is much more trustworthy.
Name two causes of false findings in published research.
Show answer
Any two of: publication bias, p-hacking, small samples.
Name three signs a finding is likely to hold up.
Show answer
Any three of: it has been replicated by independent teams, the sample was large, the effect size is plausible, the study was preregistered, it fits other evidence.
Further reading
- Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
- Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366.
- Hagger, M. S., et al. (2016). A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11(4), 546–573.
- Chabris, C. F., Heck, P. R., Mandart, J., Benjamin, D. J., & Simons, D. J. (2019). No evidence that experiencing physical warmth promotes interpersonal warmth: Two failures to replicate Williams and Bargh (2008). Social Psychology, 50(2), 127–132.
- Ranehill, E., Dreber, A., Johannesson, M., Leiberg, S., Sul, S., & Weber, R. A. (2015). Assessing the robustness of power posing: No effect on hormones and risk tolerance in a large sample of men and women. Psychological Science, 26(5), 653–656.
Finished? Mark it done to fill in your leaf on the Psychology map.