Three headlines, three studies

Three headlines appeared on the same news site in one week. The headlines and studies are made up, but each is a common kind. Work through the task yourself first, with the lessons open, then compare your answers with the solution step by step.

Headline A: "Video games make teenagers aggressive, study finds" The study: a survey of 1,200 teenagers about how many hours they play video games and how often they get into arguments or fights. More gaming went with slightly more aggression: r = 0.10, statistically significant (p < 0.001). Not repeated yet.

Headline B: "New supplement boosts memory by 30%" The study: 18 adults, paid for by the supplement's maker, took it for two weeks and did a memory test before and after. Scores rose from 10 to 13 out of 20 on average (p = 0.049). There was no control group, everyone knew what they were taking, and it has not been repeated.

Headline C: "Spacing out study sessions improves exam results, large trial confirms" The study: 1,500 students were randomly assigned to study the same material either all in one evening or spread over a week. The plan was published before the trial began. Those who spaced their study scored higher on the exam a month later (d = 0.4, p < 0.001). The result matches many earlier experiments.

Your task

1. For each headline, list what the study can and can't show, using the tools from this path (correlation and cause, experiment design, sample, significance and effect size, replication).
2. Rank the three from most to least trustworthy, with one reason each.
3. Rewrite headlines A and B so they say only what the evidence shows.

Solution

Try the task first, then open one step at a time and compare.

Step 1: headline A, the video-game survey

• Design: a survey, so it shows a link, not a cause. "Make teenagers aggressive" claims cause.
• Other explanations: a third factor (for example, less supervision at home could lead to both more gaming and more fights), or reverse direction (more aggressive teenagers may be drawn to certain games).
• Numbers: significant because the sample is large, but r = 0.10 is a weak link: gaming hours would account for about 1% of the differences in aggression between teenagers.
• Replication: not yet repeated.

Step 2: headline B, the supplement

• Design: no control group, no placebo, no blinding. Scores could rise because people practised the same test twice, because they expected to improve (placebo), or by chance.
• Sample: 18 people: small, so results swing.
• Significance: p = 0.049, only just under 0.05, from a small study: a fragile result.
• Who paid: the maker, which doesn't make it wrong, but is a reason for extra care.
• "30%": 10 to 13 is a 30% rise in score, but with no control group we can't say the supplement caused any of it.

Step 3: headline C, the spacing trial

• Design: a randomised experiment, so it can show cause, for students like these.
• Sample: 1,500, large.
• Preregistered: the plan was published first, so there was little room for p-hacking.
• Effect size: d = 0.4, between small and medium, a meaningful gain for a free change in study habits.
• Fits other evidence: it matches many earlier experiments on spacing.

Step 4: the ranking

1. C, most trustworthy: a large, preregistered, randomised experiment that fits earlier findings.
2. A, middle: a large sample, but only a weak link, and it can't show cause.
3. B, least trustworthy: tiny, no control group or placebo, paid for by the maker, and only just significant.

Step 5: honest headlines

• A: "Teens who game more report slightly more arguments and fights; the survey can't show games cause it."
• B: "Small, uncontrolled study by supplement maker reports memory gains; a proper trial is needed."

Check yourself

• I said headline A is a survey, so it shows a link, and named a third factor or reverse direction.
• I noticed r = 0.10 is weak even though it is significant.
• I named at least two other reasons headline B's scores might have risen (practice, placebo, chance).
• I said headline C can show cause because of random assignment, and noted preregistration and fit with earlier work.
• I ranked C, then A, then B, with a reason for each.
• My rewritten headlines don't claim cause.

Nothing here is graded or saved.