Study delivers bleak verdict on validity of psychology experiment results

image
Photograph: Pere Sanz / Alamy/Alamy
.
Ian Sample Science editor
.
Of 100 studies published in top-ranking journals in 2008, 75% of social psychology experiments and half of cognitive studies failed the replication test
.
There are many reasons why an experiment might fail to replicate, but more than this, the study has highlighted some issues with academic publishing and modern science.
.
Thursday 27 August 2015 14.00 EDT
.
A major investigation into scores of claims made in psychology research journals has delivered a bleak verdict on the state of the science.
.
An international team of experts repeated 100 experiments published in top psychology journals and found that they could reproduce only 36% of original findings.
.
The study, which saw 270 scientists repeat experiments on five continents, was launched by psychologists in the US in response to rising concerns over the reliability of psychology research.
.
“There is no doubt that I would have loved for the effects to be more reproducible,” said Brian Nosek, a professor of psychology who led the study at the University of Virgina. “I am disappointed, in the sense that I think we can do better.”
.
“The key caution that an average reader should take away is any one study is not going to be the last word,” he added. “Science is a process of uncertainty reduction, and no one study is almost ever a definitive result on its own.”
.
All of the experiments the scientists repeated appeared in top ranking journals in 2008 and fell into two broad categories, namely cognitive and social psychology. Cognitive psychology is concerned with basic operations of the mind, and studies tend to look at areas such as perception, attention and memory. Social psychology looks at more social issues, such as self esteem, identity, prejudice and how people interact.
.
In the investigation, a whopping 75% of the social psychology experiments were not replicated, meaning that the originally reported findings vanished when other scientists repeated the experiments. Half of the cognitive psychology studies failed the same test. Details are published in the journal Science.
.
Even when scientists could replicate original findings, the sizes of the effects they found were on average half as big as reported first time around. That could be due to scientists leaving out data that undermined their hypotheses, and by journals accepting only the strongest claims for publication.
.
Despite the grim findings, Nosek said the results presented an opportunity to understand and fix the problem. “Scepticism is a core part of science and we need to embrace it. If the evidence is tentative, you should be sceptical of your evidence. We should be our own worst critics,” he told the Guardian. One initiative now underway calls for psychologists to submit their research questions and proposed methods to probe them for review before they start their experiments.
.
John Ioannidis, professor of health research and policy at Stanford University, said the study was impressive and that its results had been eagerly awaited by the scientific community. “Sadly, the picture it paints – a 64% failure rate even among papers published in the best journals in the field – is not very nice about the current status of psychological science in general, and for fields like social psychology it is just devastating,” he said.
.
But he urged people to focus on the positives. The results, he hopes, will improve research practices in psychology and across the sciences more generally, where similar problems of reproducibility have been found before. In 2005, Ioannidis published a seminal study that explained why most published research findings are false.
.
Marcus Munafo, a co-author on the study and professor of psychology at Bristol University, said: “I think it’s a problem across the board, because wherever people have looked, they have found similar issues.” In 2013, he published a report with Ioannidis that found serious statistical weaknesses were common in neuroscience studies.
.
Continued in comment section
.
.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: