Experiments give different results second time around

NEW YORK — A large group of researchers set out to repeat 100 experiments published by leading psychology journals to see how often they would get the same results. The answer was less than half the time.

That doesn’t mean all those unconfirmed studies were wrong.

But it’s a stark reminder that a single study rarely provides definitive answers and why scientists often greet new findings by saying: “More research is needed.”

“Any one study is not going to be the last word,” said Brian Nosek, a psychology professor at the University of Virginia.

“Each individual study has some evidence. It contributes some information toward a conclusion. But the real conclusion, when you can say confidently that something is true or false, is based on an accumulation of evidence over many studies,” said Nosek, who led the project.

He told a news conference that “even this project itself is not a definitive word about reproducibility.”

The work was carried out by an international team of more than 300 people and released this week by the journal Science. The project focused on psychology because its organizers came from that field.

Researchers worked with the authors of the original studies in setting up the replication attempts.

Only about 40 per cent of those attempts produced the original results.

The effort focused on 100 experiments reported during 2008 in any of three major psychology journals: Psychological Science, the Journal of Personality and Social Psychology and the Journal of Experimental Psychology: Learning, Memory, and Cognition.

None of these experiments tested any treatments. They focused on basic research into how people think, remember, perceive their world and interact with others.

One explored why people are reluctant to tempt fate, for example.

Studies with stronger statistical evidence for their conclusions were more likely to be replicated than others, as were those with findings that were judged to be less surprising.